AutoML-Zero: Evolving Machine Learning Algorithms From Scratch
AutoML-Zero uses evolutionary search to discover complete ML algorithms from basic math operations, surpassing traditional AutoML limitations.
Key Findings
Methodology
AutoML-Zero employs an evolutionary framework representing algorithms as programs with three components: Setup, Predict, and Learn. These programs are constructed solely from fundamental mathematical operations like addition, multiplication, trigonometric functions, etc. The search involves random mutations and tournament selection, guided by performance on proxy tasks. Functional equivalence detection prevents redundant evaluations, significantly speeding up the process. Starting from an empty program, the system evolves algorithms capable of training neural networks, discovering techniques like bilinear interactions, normalized gradients, and weight averaging. Experiments demonstrate successful evolution of algorithms that outperform traditional gradient-based training on CIFAR-10 variants, with adaptability to different data regimes and tasks.
Key Results
- The evolved algorithms on CIFAR-10 variants outperform baseline neural networks trained by gradient descent, achieving higher accuracy and demonstrating the emergence of modern techniques such as bilinear interactions and gradient normalization.
- Compared to random search, the evolutionary method is over five times more efficient, successfully discovering complex algorithms from an enormous, sparse search space.
- The algorithms adapt to task-specific needs, with dropout-like regularization appearing in low-data scenarios and learning rate decay in faster convergence tasks, indicating strong generalization and flexibility.
Significance
This work fundamentally shifts the paradigm of AutoML by demonstrating the feasibility of discovering complete algorithms from scratch without human-designed modules. It addresses core limitations of existing approaches that rely heavily on expert knowledge and constrained search spaces. By enabling the automatic synthesis of algorithms, including those beyond neural networks, it opens new avenues for innovation in machine learning. The approach reduces human bias, accelerates discovery, and offers a pathway toward fully autonomous AI systems capable of evolving novel solutions tailored to diverse tasks, thus holding profound implications for both research and industry.
Technical Contribution
The paper introduces a novel program-based representation of algorithms, combined with a scalable evolutionary search enhanced by functional equivalence detection and distributed migration. This allows the automatic discovery of complex learning algorithms from a minimal set of basic operations. The framework supports the evolution of entire training procedures, initialization schemes, and model architectures, broadening the scope of AutoML beyond architecture search. The open-source implementation and extensive experimental validation establish a new benchmark for algorithm synthesis, emphasizing minimal human bias and maximal automation.
Novelty
This is the first work to demonstrate the automatic discovery of complete machine learning algorithms solely from basic mathematical operations, starting from an empty program. Unlike existing AutoML methods that focus on neural architecture search or hyperparameter tuning within constrained spaces, AutoML-Zero explores an unbounded, sparse search space, enabling the emergence of innovative algorithms and techniques. Its ability to evolve algorithms capable of training neural networks and adapting to different tasks from scratch marks a significant breakthrough in the field.
Limitations
- The search process requires enormous computational resources and time, limiting accessibility and scalability in resource-constrained environments.
- The discovered algorithms, while effective, are often complex and lack interpretability, posing challenges for understanding and debugging.
- Current validation is limited to small-scale tasks like CIFAR-10 variants; effectiveness on large-scale, real-world problems remains to be demonstrated.
Future Work
Future research will focus on improving search efficiency through advanced optimization techniques, integrating reinforcement learning, and reducing computational costs. Extending the framework to larger, more complex datasets and models, enhancing interpretability, and ensuring robustness are key directions. Additionally, combining hardware-aware search and meta-learning could further accelerate practical deployment, ultimately aiming for fully autonomous systems capable of continuous self-improvement.
AI Executive Summary
AutoML-Zero pioneers a transformative approach in automated machine learning by demonstrating that complete algorithms can be evolved from scratch using only fundamental mathematical operations. Traditional AutoML methods primarily focus on neural architecture search or hyperparameter tuning within predefined, human-designed spaces, which inherently limits innovation and introduces human bias. In contrast, AutoML-Zero constructs a vast, unstructured search space where algorithms are represented as small programs composed of basic instructions. Using evolutionary strategies, these programs undergo mutation and selection, guided by performance on proxy tasks, to discover effective learning algorithms.
The core innovation lies in representing algorithms as sequences of instructions with three key components: Setup, Predict, and Learn. These components are built solely from simple operations like addition, multiplication, and trigonometric functions, enabling the discovery of novel training procedures. The framework incorporates functional equivalence detection to avoid redundant evaluations, significantly speeding up the search process. Experiments show that starting from an empty program, the system can evolve algorithms capable of training neural networks on CIFAR-10 variants, surpassing baseline methods and even discovering modern techniques such as bilinear interactions, gradient normalization, and weight averaging.
Beyond simple tasks, the evolved algorithms adapt to different data regimes, introducing dropout-like regularization when data is scarce and learning rate decay for faster convergence. These results demonstrate the potential of fully automated algorithm discovery, opening new avenues for research beyond neural networks. While computationally intensive, this approach offers a promising path toward autonomous AI systems capable of continuous self-improvement, with broad implications for both academia and industry. Future work will aim to enhance efficiency, interpretability, and scalability, pushing the boundaries of what automated algorithm synthesis can achieve.
Deep Analysis
Background
近年来，深度学习在图像识别、自然语言处理等领域取得了巨大突破，推动神经网络技术不断演进。代表性工作包括Hinton的深度置信网络、LeCun的卷积神经网络等。随着模型复杂度提升，架构搜索（NAS）和超参数调优成为瓶颈，促使AutoML应运而生，旨在自动化这些繁琐过程。早期AutoML多依赖预定义模块，限制了创新空间。近年来，神经架构搜索取得一定成功，但仍受限于设计的搜索空间。本文背景是希望突破这些限制，探索从基础数学操作到完整算法的自动发现，推动AutoML走向更自主、更创新的未来。
Core Problem
现有AutoML方法多依赖于有限的搜索空间和专家知识，限制了算法的多样性和创新性。神经网络架构和训练策略的自动化虽取得进展，但缺乏从零开始的全自动算法发现能力。核心难题在于如何在庞大且稀疏的搜索空间中高效找到具有竞争力的算法，尤其是无需预设模型结构或梯度信息。解决这一问题对于推动AutoML的自主性和创新能力具有重要意义，但面临搜索效率、算法复杂性和可解释性等挑战。
Innovation
本研究的创新在于提出AutoML-Zero框架，利用演化搜索在极其庞大且无偏的搜索空间中自动发现算法。其核心创新点包括：• 将算法表示为由基础数学操作组成的程序，避免人为偏见；• 引入功能等价检测，显著提升搜索效率；• 通过迁移和代理任务，增强算法的泛化能力；• 从空程序开始演化，逐步生成训练神经网络的完整流程。这些创新突破了以往AutoML对结构和模块的依赖，开启了算法自主发现的新纪元。
Methodology
- �� 定义算法为由Setup、Predict、Learn三部分组成的程序，全部由基础数学操作构成；• 利用演化算法，通过随机突变和选择优化程序性能，适应不同任务；• 采用功能等价检测，避免重复评估，加快搜索速度；• 使用代理任务和迁移技术，提升搜索效率和算法泛化能力；• 从空程序开始，逐步演化出训练神经网络的完整算法，涵盖模型初始化、训练和预测流程；• 在不同任务（如CIFAR-10变体）上优化，发现现代深度学习技术。
Experiments
实验在CIFAR-10及其变体上进行，采用演化搜索和随机搜索对比。代理任务通过低维投影加快评估速度，最终在原始数据上验证算法性能。评估指标为分类准确率，使用中位数作为算法质量指标。实验中，从空程序开始，经过多轮演化，成功发现训练神经网络的算法，超越手工设计的基线。还在不同数据量和任务类型下验证算法的适应性，发现正则化和学习率调节等现代技术。实验耗时数天，利用迁移和分布式技术提升搜索效率，验证了方法的可行性和有效性。
Results
演化算法在CIFAR-10变体任务中成功发现了性能优于梯度训练神经网络的算法，准确率提升显著。算法中出现了双线性交互、归一化梯度和权重平均等现代深度学习技术，验证了其创新能力。与随机搜索相比，演化搜索效率提升了5倍以上。算法还能根据任务需求自动调节，出现dropout样的正则化和学习率衰减，表现出良好的适应性。这些结果证明了从零到完整算法的自动演化潜力，为AutoML的未来提供了新方向。
Applications
该方法可用于自动生成适应不同任务的机器学习算法，特别适合缺乏专家知识或需要创新的场景。未来可结合硬件感知优化，应用于边缘设备、自动驾驶、机器人等领域，实现自主算法设计，降低人工成本，提升系统智能化水平。
Limitations & Outlook
当前方法依赖大量计算资源，搜索过程耗时长，难以在资源有限环境中普及。算法的可解释性较差，自动生成的算法难以理解和调试。尚未在大规模深度模型和复杂任务中验证其效果，未来需优化搜索效率和模型复杂度，增强可解释性和鲁棒性。
Plain Language Accessible to non-experts
想象一个工厂，没有预制的机器或流程，工人只用最基础的工具（锤子、钳子、剪刀）设计制造新机器。每次试错后，他们根据效果选择更好的设计，不断改进。经过反复尝试，工厂最终能制造出复杂的自动装配线或机器人。AutoML-Zero就像这个工厂，用最基本的数学工具，通过演化和筛选，自动“设计”出能完成特定任务的算法。这种方法不依赖预设模板，而是让算法自己“学习”如何学习，逐步变得更聪明。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，没有提示，也没有预拼好的模型。你只有一些简单的拼图块，比如直线、圆形、三角形。你试着用这些基础块拼出一幅画，但一开始拼得乱七八糟。然后，你不断尝试不同的拼法，挑选那些看起来更像样的，舍弃不好的方案。经过很多次努力，你终于拼出了一个漂亮的图像。AutoML-Zero就像这个游戏，用最简单的数学操作，反复试验和挑选，最终“拼”出能解决问题的算法。整个过程由电脑自己完成，不需要人提前设计每一步，就像让电脑自己学会拼图一样。
Glossary
AutoML (自动机器学习)
自动化设计和优化机器学习模型的技术，减少人工干预，提高效率。In this paper, AutoML refers to the automated discovery of algorithms using evolutionary search.
描述自动化算法发现的整体框架。
演化搜索 (Evolutionary Search)
一种基于自然选择原理的优化方法，通过突变和选择逐步改进候选方案。In this paper, used to自动发现算法。
核心搜索策略，用于在庞大空间中优化算法程序。
功能等价检测 (Functional Equivalence Checking)
判断两个程序是否在所有输入下表现一致的方法，以避免重复评估。In this paper,提升搜索效率。
用于加速演化过程，避免重复计算。
代理任务 (Proxy Tasks)
简化的任务，用于快速评估候选算法性能。In this paper,用于加快搜索。
在搜索阶段快速筛选潜在算法。
基础数学操作 (Basic Mathematical Operations)
加减乘除、三角函数、指数等最基本的计算指令。In this paper,构成算法程序的基本单元。
构建搜索空间的核心元素。
Open Questions Unanswered questions from this research
- 1 如何进一步提升搜索效率，减少计算成本，特别是在复杂任务和大规模模型上实现自动算法发现的可行性仍未解决。
- 2 自动生成算法的可解释性和可靠性问题，如何确保自动设计的算法在实际应用中的安全性和稳定性。
Applications
Immediate Applications
自动算法设计工具
可用于科研和工业界，自动生成适应不同任务的机器学习算法，减少人工调优时间，提升创新能力。
快速原型开发
在新兴应用场景中，快速生成有效算法，缩短开发周期，适应多变需求。
Long-term Vision
自主智能系统
未来实现完全自主设计和优化算法的智能系统，减少人类干预，推动AI自主创新。
Abstract
Machine learning research has advanced in multiple aspects, including model structures and learning methods. The effort to automate such research, known as AutoML, has also made significant progress. However, this progress has largely focused on the architecture of neural networks, where it has relied on sophisticated expert-designed layers as building blocks---or similarly restrictive search spaces. Our goal is to show that AutoML can go further: it is possible today to automatically discover complete machine learning algorithms just using basic mathematical operations as building blocks. We demonstrate this by introducing a novel framework that significantly reduces human bias through a generic search space. Despite the vastness of this space, evolutionary search can still discover two-layer neural networks trained by backpropagation. These simple neural networks can then be surpassed by evolving directly on tasks of interest, e.g. CIFAR-10 variants, where modern techniques emerge in the top algorithms, such as bilinear interactions, normalized gradients, and weight averaging. Moreover, evolution adapts algorithms to different task types: e.g., dropout-like techniques appear when little data is available. We believe these preliminary successes in discovering machine learning algorithms from scratch indicate a promising new direction for the field.