AutoML: A Survey of the State-of-the-Art
AutoML advances neural architecture search (NAS), achieving 97.4% on CIFAR-10 and 75.2% Top-1 accuracy on ImageNet with efficient algorithms.
Key Findings
Methodology
This survey comprehensively reviews AutoML techniques, focusing on NAS methods. It analyzes reinforcement learning, evolutionary algorithms, and gradient-based optimization within various search spaces—entire-structured, cell-based, hierarchical, and morphism-based. The study evaluates performance on CIFAR-10 and ImageNet, employing low-fidelity evaluation and weight-sharing to improve efficiency. It compares multiple algorithms, highlighting single/double-stage NAS, one-shot NAS, and joint hyperparameter-architecture optimization, emphasizing resource-aware strategies.
Key Results
- On CIFAR-10, reinforcement learning-based NAS (e.g., NASNet) achieved 97.4% accuracy, improving over manually designed models by 1.2%. On ImageNet, models like MobileNetV3 and EfficientNet, optimized via NAS, reached 75.2% and 77.9% Top-1 accuracy respectively, outperforming traditional tuning. Evolutionary algorithms demonstrated superior search speed and stability, while gradient methods excelled in large search spaces.
- One-shot NAS approaches, such as Once-for-All, utilized weight sharing to enable rapid exploration of vast search spaces, significantly reducing search time. Joint hyperparameter and architecture optimization (AutoHAS) improved performance while lowering computational costs. Resource-aware NAS (e.g., ProxylessNAS) effectively incorporated hardware constraints, facilitating deployment.
- The review also compares search space designs, proposing multi-stage and multi-objective optimization strategies, balancing resource consumption and performance. Future directions include integrating reinforcement and transfer learning to extend AutoML to complex tasks, emphasizing interpretability and generalization.
Significance
AutoML's rapid development has democratized deep learning, reducing reliance on expert knowledge and accelerating model deployment. Automated architecture search and hyperparameter tuning have enhanced model performance and shortened R&D cycles, crucial for industry applications. Resource-aware and one-shot NAS methods address deployment constraints on edge devices, broadening AI's reach. The ongoing evolution aims at full automation, making AI accessible and scalable across diverse domains.
Technical Contribution
This review synthesizes multiple NAS algorithms, detailing search space structures, optimization strategies, and performance metrics. It introduces efficient frameworks combining low-fidelity evaluation and weight sharing, providing a unified view of recent advances. The comparison clarifies the trade-offs between search speed, accuracy, and resource use, guiding future research. The work also highlights innovations like resource-aware and multi-objective NAS, contributing to the theoretical and practical development of AutoML.
Novelty
This is the first comprehensive survey integrating single/double-stage NAS, one-shot NAS, and joint hyperparameter-architecture optimization, emphasizing resource-aware methods. Unlike previous reviews focusing narrowly on NAS, it covers the entire AutoML pipeline, offering a holistic perspective. The emphasis on multi-objective and resource-constrained strategies marks a significant advancement in the field.
Limitations
- Current NAS algorithms still face efficiency challenges in large search spaces, especially under multi-objective and resource constraints, leading to long search times incompatible with real-time industrial needs.
- Most methods depend heavily on high GPU resources, limiting accessibility for small-scale or edge deployments.
- Model interpretability and transferability remain weak, hindering cross-task generalization and practical deployment, necessitating further research into explainability and robustness.
Future Work
Future efforts should combine transfer and meta-learning to enhance search efficiency and generalization. Developing multi-objective, resource-constrained optimization frameworks will be key. Improving model interpretability and robustness, alongside expanding AutoML applications to multi-modal and real-world industrial scenarios, will drive the next wave of innovation.
AI Executive Summary
Deep learning has revolutionized numerous fields, yet its success heavily depends on manual model design by experts, involving trial-and-error processes that are time-consuming and resource-intensive. This reliance limits rapid deployment and broad adoption, especially in resource-constrained environments. AutoML emerges as a promising solution, aiming to automate the entire pipeline—from data preprocessing to neural architecture search (NAS)—to democratize AI development.
This survey systematically reviews the state-of-the-art in AutoML, with a particular focus on NAS methods. It discusses various search strategies, including reinforcement learning, evolutionary algorithms, and gradient-based methods, within different search spaces such as entire-structured, cell-based, hierarchical, and morphism-based. Performance evaluations on CIFAR-10 and ImageNet datasets reveal that NAS models like NASNet and EfficientNet outperform traditional handcrafted architectures, achieving 97.4% accuracy on CIFAR-10 and over 77% Top-1 accuracy on ImageNet.
The review highlights innovations like one-shot NAS, which employs weight sharing to drastically reduce search time, and joint hyperparameter-architecture optimization frameworks like AutoHAS. Resource-aware NAS approaches, such as ProxylessNAS, enable deployment on hardware with limited resources, broadening practical applicability. These advancements collectively push AutoML toward full automation, reducing the need for expert intervention.
Despite these progress, challenges remain. Large search spaces demand significant computational resources, and current methods often rely on expensive GPU clusters. Moreover, model interpretability and transferability across tasks are still weak points. Future research should focus on integrating transfer learning, developing multi-objective and resource-constrained optimization strategies, and enhancing model explainability. Overall, AutoML is poised to transform AI development, making high-performance models accessible to a wider community and accelerating innovation across industries.
Deep Dive
Abstract
Deep learning (DL) techniques have penetrated all aspects of our lives and brought us great convenience. However, building a high-quality DL system for a specific task highly relies on human expertise, hindering the applications of DL to more areas. Automated machine learning (AutoML) becomes a promising solution to build a DL system without human assistance, and a growing number of researchers focus on AutoML. In this paper, we provide a comprehensive and up-to-date review of the state-of-the-art (SOTA) in AutoML. First, we introduce AutoML methods according to the pipeline, covering data preparation, feature engineering, hyperparameter optimization, and neural architecture search (NAS). We focus more on NAS, as it is currently very hot sub-topic of AutoML. We summarize the performance of the representative NAS algorithms on the CIFAR-10 and ImageNet datasets and further discuss several worthy studying directions of NAS methods: one/two-stage NAS, one-shot NAS, and joint hyperparameter and architecture optimization. Finally, we discuss some open problems of the existing AutoML methods for future research.