Towards Learning Universal Hyperparameter Optimizers with Transformers
OptFormer, a Transformer-based universal hyperparameter optimizer, can imitate 7 algorithms, leveraging large-scale real tuning data to improve prediction and optimization.
Key Findings
Methodology
This paper introduces OptFormer, employing a T5 encoder-decoder architecture to serialize hyperparameter studies into text sequences. The model jointly learns policy and function prediction, trained on vast real-world tuning data such as Google Vizier. It mimics multiple algorithms (e.g., random search, evolutionary, Bayesian UCB) by modeling their trajectories, capturing diverse strategies. The training maximizes conditional log-likelihood, predicting hyperparameter suggestions and response functions. Uncertainty estimates are integrated to enhance sampling strategies, with acquisition functions like Expected Improvement (EI) used for online optimization. The approach enables multi-task, multi-space generalization, surpassing traditional Gaussian Processes in predictive accuracy and calibration.
Key Results
- OptFormer successfully imitates at least 7 different HPO algorithms, with trajectories closely matching real algorithms. It outperforms Gaussian Processes in log-likelihood (2.12 vs. 0.83) and calibration (ECE 1.11% vs. 5.34%) on large-scale datasets, demonstrating superior predictive quality.
- On Google’s real-world database (~750K studies), the model exhibits strong generalization, accurately predicting hyperparameter suggestions and response functions, enabling effective policy transfer across diverse tasks.
- Incorporating acquisition functions like EI, OptFormer achieves competitive or superior tuning performance compared to traditional Bayesian optimization, especially in complex, multi-task scenarios, validating its practical utility.
Significance
This work advances hyperparameter optimization by leveraging Transformer architectures to learn flexible, multi-strategy, multi-space policies from massive real-world data. It addresses the limitations of fixed-space Bayesian methods, offering a unified framework capable of modeling complex behaviors and uncertainties. This innovation significantly impacts AutoML, enabling more efficient, scalable, and adaptive hyperparameter tuning, with broad industrial applications. It also opens new research avenues in meta-learning, sequence modeling, and probabilistic inference within black-box optimization. The approach bridges the gap between deep learning and classical Bayesian methods, fostering a new paradigm for intelligent model tuning.
Technical Contribution
The paper pioneers a Transformer-based framework for black-box hyperparameter optimization, transforming the problem into sequence modeling via serialization. It supports multi-algorithm imitation, joint policy and function learning, and uncertainty estimation, enabling strategy augmentation with acquisition functions. The training on large-scale datasets demonstrates the model’s ability to generalize across diverse search spaces and algorithms, surpassing Gaussian Processes in predictive likelihood and calibration. This approach introduces a novel way to integrate deep sequence models into Bayesian optimization workflows, expanding the scope and flexibility of automated hyperparameter tuning.
Novelty
This is the first work applying Transformer architectures to black-box HPO, enabling multi-strategy imitation and joint policy-function modeling across different search spaces. Unlike prior fixed-space or single-algorithm methods, it serializes heterogeneous data into text, allowing flexible input handling. The integration of uncertainty estimation and acquisition functions within a Transformer framework is also novel, opening new directions for adaptive, data-driven optimization. This represents a significant departure from traditional GP-based methods, offering a scalable, expressive alternative capable of leveraging massive real-world datasets.
Limitations
- Despite strong performance, the model’s accuracy diminishes on highly complex or noisy functions, especially with limited data. Its reliance on large datasets poses challenges for low-resource scenarios.
- Computational costs for inference and training remain high, particularly in high-dimensional spaces, limiting real-time applications.
- Transferability to unseen search spaces or algorithms requires further validation, and the model may need fine-tuning for specific domains.
Future Work
Future research will focus on scaling the model to higher-dimensional spaces, integrating reinforcement learning for adaptive online strategies, and reducing data dependency through transfer learning. Exploring model compression and efficient inference techniques will make deployment more feasible. Additionally, extending the framework to multi-objective and constrained optimization tasks can broaden its applicability, ultimately aiming for a fully autonomous, general-purpose hyperparameter optimizer.
AI Executive Summary
Hyperparameter tuning remains a critical bottleneck in machine learning, often relying on heuristic or fixed-space Bayesian methods like GP-UCB. While effective in controlled settings, these approaches struggle with complex, multi-task, and high-dimensional problems. Recent advances in deep learning, especially Transformer architectures, have revolutionized sequence modeling, inspiring new approaches in AutoML. This paper introduces OptFormer, a Transformer-based framework capable of learning a unified policy for hyperparameter optimization across diverse algorithms and search spaces.
By serializing hyperparameter studies into text sequences, OptFormer leverages large-scale real-world data from Google Vizier, encompassing over 750,000 studies. The model jointly predicts hyperparameter suggestions and response functions, capturing complex behaviors and uncertainties. It can imitate multiple algorithms simultaneously, such as random search, evolutionary strategies, and Bayesian UCB, demonstrating high fidelity in trajectory reproduction and predictive calibration.
Experimental results show that OptFormer surpasses Gaussian Processes in predictive likelihood and calibration, providing more reliable uncertainty estimates. When combined with acquisition functions like Expected Improvement, it achieves competitive or superior optimization performance in benchmark tasks, including synthetic functions (BBOB) and real-world tuning scenarios. These findings highlight the potential of deep sequence models to revolutionize AutoML by enabling scalable, flexible, and data-driven hyperparameter tuning.
Looking ahead, future work aims to extend the model’s capacity to higher-dimensional spaces, incorporate reinforcement learning for adaptive online strategies, and reduce data requirements. The integration of Transformer-based models into HPO workflows marks a significant step toward fully autonomous, intelligent model tuning systems, promising broad impact across industry and academia.
Deep Dive
Abstract
Meta-learning hyperparameter optimization (HPO) algorithms from prior experiments is a promising approach to improve optimization efficiency over objective functions from a similar distribution. However, existing methods are restricted to learning from experiments sharing the same set of hyperparameters. In this paper, we introduce the OptFormer, the first text-based Transformer HPO framework that provides a universal end-to-end interface for jointly learning policy and function prediction when trained on vast tuning data from the wild, such as Google's Vizier database, one of the world's largest HPO datasets. Our extensive experiments demonstrate that the OptFormer can simultaneously imitate at least 7 different HPO algorithms, which can be further improved via its function uncertainty estimates. Compared to a Gaussian Process, the OptFormer also learns a robust prior distribution for hyperparameter response functions, and can thereby provide more accurate and better calibrated predictions. This work paves the path to future extensions for training a Transformer-based model as a general HPO optimizer.