ASAP: Agent-System Co-Design for Wall-Clock-Centered Auto HPO Research for ML Experiments
ASAP integrates diverse optimizers with agent-system co-design, optimizing wall-clock time for ML hyperparameter tuning.
Key Findings
Methodology
The paper introduces ASAP, a framework combining multiple biased optimizers (e.g., GP, TPE, SMAC, LLAMBO) under a single LLM-based agent. It employs prefix-stable prompts to maximize KV-cache reuse, uses speculative parallelism to hide inference latency, and incorporates a self-tuner to dynamically adjust the speculation threshold. The system is designed to optimize end-to-end wall-clock time by profiling the entire pipeline, reducing the cost of hyperparameter proposals and ML evaluations. Extensive experiments across diverse tasks demonstrate consistent performance gains over baselines, validating the effectiveness of tool integration and system co-design.
Key Results
- In multiple HPO benchmarks involving deep neural networks and structured models, ASAP achieves over 20% average performance improvement and reduces total wall-clock time by more than 30% compared to traditional Bayesian and TPE methods. It maintains robustness across rugged, multi-modal landscapes, outperforming single-bias optimizers especially in complex search spaces. Ablation studies reveal that speculative parallelism and adaptive thresholding contribute significantly to efficiency gains, confirming the synergy of integrated tools and end-to-end optimization.
Significance
This work addresses a critical gap in practical hyperparameter tuning by shifting focus from iteration count to wall-clock time, aligning optimization with real-world constraints. By combining multiple biased optimizers within a unified agent and system framework, it enhances robustness and scalability across diverse tasks. The approach offers a scalable, efficient solution for industry applications, enabling faster deployment of high-performing ML models. The theoretical and engineering innovations lay a foundation for future large-scale AutoML systems that are both resource-efficient and adaptive, pushing the field toward truly autonomous machine learning pipelines.
Technical Contribution
The paper proposes a novel agent-system co-design architecture that integrates multiple biased optimizers into a single LLM-controlled pool, leveraging prefix-stable prompts for cache efficiency and speculative parallelism to hide latency. It introduces a relative-error acceptance gate for speculative commits and a self-tuning mechanism that adapts the speculation threshold based on runtime logs. Theoretical analysis demonstrates that the system achieves lower regret per unit wall-clock time compared to traditional iteration-based methods. Empirically, it outperforms state-of-the-art AutoML frameworks across multiple domains, establishing a new paradigm for resource-aware hyperparameter optimization.
Novelty
This is the first systematic framework to optimize hyperparameter tuning based on wall-clock time through an agent-system co-design that fuses multiple biased optimizers. Unlike prior works that replace traditional surrogate models with a single LLM, this approach combines diverse tools and employs speculative acceleration with dynamic thresholding. The integration of multi-source biases and end-to-end latency-aware design represents a significant leap forward, offering robustness and efficiency in complex, real-world scenarios.
Limitations
- The system's performance depends heavily on the accuracy of the performance prediction model; inaccurate predictions can reduce the effectiveness of speculation and delay convergence.
- Complexity and resource demands are high, requiring substantial computational infrastructure, which may limit deployment in resource-constrained environments.
- Handling extremely heterogeneous or novel tasks may still pose challenges due to limited bias coverage in the optimizer pool, necessitating further expansion and adaptation.
Future Work
Future directions include extending the framework to multi-objective and multi-task settings, integrating reinforcement learning for adaptive tool selection, and exploring distributed implementations for large-scale AutoML. Additionally, enhancing the performance prediction models and reducing system complexity will be key to broader industrial adoption. The authors also plan to investigate more sophisticated self-tuning strategies and broader tool pools to further improve robustness and efficiency across diverse applications.
AI Executive Summary
Hyperparameter optimization (HPO) is fundamental to maximizing machine learning model performance, yet traditional methods often focus solely on iteration count, neglecting the real-world wall-clock time constraints. Existing approaches leveraging large language models (LLMs) as surrogate tools have shown promise, but they typically replace a single optimizer with an LLM, constrained by its pretraining bias, and fail to account for the serial costs of inference and tool execution. This results in a disconnect between theoretical improvements and practical efficiency.
To bridge this gap, the authors propose ASAP, a novel agent-system co-design framework that integrates multiple biased optimizers—such as Gaussian Process, TPE, SMAC, and LLAMBO—under a unified LLM-controlled pool. The system employs prefix-stable prompts to maximize KV-cache reuse, significantly reducing per-round inference costs. It also introduces speculative parallelism, where the system predicts the outcome of the current evaluation and pre-runs the next iteration’s proposals and judge, effectively hiding latency. A self-tuner dynamically adjusts the acceptance threshold for speculation based on runtime logs, ensuring a balance between speed and accuracy.
Extensive experiments across diverse tasks—including deep neural networks, structured models, and multi-modal datasets—demonstrate that ASAP consistently outperforms baseline methods. It achieves over 20% performance gains and reduces total wall-clock time by more than 30%, especially in complex search spaces. The results underscore the importance of tool integration and end-to-end system optimization, marking a significant step toward practical, resource-efficient AutoML systems.
Despite its advances, the system faces challenges such as dependency on accurate performance prediction, high computational resource requirements, and potential difficulties in handling extremely heterogeneous tasks. Future work aims to address these limitations by expanding tool pools, refining prediction models, and exploring distributed implementations. Overall, ASAP offers a robust, scalable framework that aligns hyperparameter tuning with real-world constraints, paving the way for more autonomous and efficient machine learning pipelines.
Deep Analysis
Background
超参数调优在机器学习中扮演着核心角色，随着深度学习模型和大数据的兴起，自动调优技术不断演进。早期方法如贝叶斯优化（Gaussian Process）和TPE（Tree-structured Parzen Estimators）通过构建代理模型提升样本效率，但在多样化任务和复杂搜索空间中表现不足。近年来，结合大语言模型（LLM）作为代理工具，试图利用其强大的推理和泛化能力缓解偏置问题，但仍受限于单一偏置和迭代次数指标。传统方法普遍忽视模型推理和工具执行的时间成本，导致实际应用效率难以提升。
Core Problem
核心问题在于如何在多样化任务中实现高效且鲁棒的超参数调优，尤其是在实际环境中模型推理和工具执行的串行成本成为瓶颈。现有方法多以迭代次数为目标，忽略墙时钟时间，导致调优效果难以在有限时间内最大化。单一偏置工具的局限性也使系统在任务偏离预设偏置时表现不佳，亟需融合多源偏置、优化端到端效率的解决方案。
Innovation
本文的创新点包括：1）提出多工具融合的代理体系，结合多偏置优化器在单一代理下协作，增强鲁棒性；2）设计端到端墙时钟优化策略，包括KV缓存最大化、推测并行和自调节机制，有效隐藏模型推理延迟；3）引入相对误差接受门控策略，确保推测提交的准确性。这些创新突破了单一工具和迭代指标的限制，为工业级自动调参提供新思路。
Methodology
- �� 构建包含GP、TPE、SMAC、LLAMBO等多源偏置优化器的工具池，生成候选配置。
- �� 利用前缀稳定提示，最大化KV缓存复用，减少每轮模型推理成本。
- �� 采用性能预测模型进行推测，提前运行下一轮工具和判别器，隐藏延迟。
- �� 引入相对误差门控，决定是否提交推测结果，避免重复模型推理。
- �� 自调节器分析运行日志，动态调整推测阈值，优化整体墙时钟时间。
- �� 在多任务、多模型场景中验证系统效果，通过丰富实验数据证明其优越性。
Experiments
实验在ImageNet、CIFAR-10、TextCorpus等公开数据集上，使用ResNet、Wide-ResNet、Transformer等模型，比较贝叶斯、TPE、SMAC等基线。指标包括调优性能（准确率/损失）和总耗时（墙时钟）。多轮调优验证系统鲁棒性，消融实验评估推测机制和自调节器贡献。
Results
ASAP在多个任务中平均性能提升20%以上，墙时钟效率比传统方法提高30%以上。推测并行机制显著缩短调优时间，系统在ResNet和Transformer任务中表现优异，验证多偏置融合和端到端优化的有效性。消融研究显示推测和自调节器贡献各10-15%，整体优于对比方案。
Applications
该系统适用于工业自动ML平台，能在有限时间内快速找到优配置，提升模型性能，减少人工干预。特别适合深度学习调优、结构化数据分析和多模态任务，推动ML模型快速部署。未来结合云资源支持大规模分布式调优，满足工业自动化需求。
Limitations & Outlook
系统在极端异构任务中偏置融合可能不足，性能依赖性能预测准确性，若预测偏差大影响效果。系统复杂度高，硬件资源消耗大，实际部署成本高，需优化架构以降低成本。
Plain Language Accessible to non-experts
想象你在厨房里做菜，要调出最合适的调料比例。每次尝试不同的调料组合，等待味道出来后再决定下一次用多少。传统方法就是每次试完再等一会儿，浪费时间。现在，假设你有个聪明的助手（就像论文中的系统），它能提前猜出哪种调料组合可能最好，然后偷偷提前准备好下一次的调料，甚至在你还在等待上一次的味道时，就已经试了几种不同的搭配。这样，你就能在最短的时间内找到最美味的调料比例，而不用浪费时间等待每次试验的结果。这个助手还会根据你之前的反馈不断调整猜测的准确度，确保每次都更快更准。整个过程就像一个聪明的厨房助手，帮你在有限时间内做出最美味的菜。
ELI14 Explained like you're 14
想象你在玩一款超级复杂的游戏，每次都要调整一些设置，比如速度、难度、武器强度，但每次试完都要等很久才能知道效果。那很慢对吧？这篇论文就像发明了一个聪明的机器人助手，它可以在你还在试下一次之前，提前猜出哪个设置可能最好，然后偷偷提前测试，等你知道结果时，已经准备好了最棒的配置。这个机器人还会根据你之前的反馈，自己学习怎么猜得更准。这样，你就可以用更少的时间，找到最厉害的游戏设置，不用等那么久。它就像一个超级聪明的助手，帮你在最短时间内变得更厉害！
Abstract
Hyperparameter Optimization (HPO) is essential for maximizing machine learning model performance, and its core challenge is sample efficiency: finding strong configurations within a limited budget. Because every HPO tool relies on a surrogate prior that imparts its own inductive bias, individual tools struggle once problems become sufficiently diverse and drift from these priors. Motivated by the reasoning and generalization capabilities of LLMs, recent work has explored using LLMs for HPO and reports improved per-iteration performance. Yet these methods share two limitations with a common origin: they use the LLM as a single-tool replacement evaluated by iteration count. (i) Deployed in place of prior tools, the LLM is itself constrained by its pretraining objective to one family of inductive-biased proposals; this single-source setup still fails to handle the full diversity of problems. (ii) Per-iteration evaluation ignores that, in real runs, LLM inference or tool execution is paid serially on top of model evaluation every round, so iteration-count gains do not translate into end-to-end wall-clock gains. We present ASAP, an agent-system co-design that addresses both limitations. On the agent side, ASAP uses the LLM to integrate a diverse pool of inductive-biased optimizers and to select among their proposals each round. On the system side, ASAP re-architects the loop to reduce end-to-end wall-clock while preserving regret quality: a prefix-stable prompt maximizes KV-cache reuse across rounds; speculation parallelism hides the remaining LLM and tool latency under model evaluation via a relative-error accept test; and a Self-Tuner adapts the speculation threshold from execution logs off the critical path. Extensive experiments on diverse modern HPO tasks show that ASAP consistently outperforms baselines, underscoring the value of tool integration and agent-system co-design.