Blog: Survey of Optimizers
This paper systematically organizes 2025-2026 optimizers into a multi-dimensional framework, emphasizing matrix-awareness, temporal estimation, and system representation, highlighting no single optimizer dominates.
Key Findings
Methodology
The paper classifies optimizers along four largely independent axes: temporal estimation, update geometry, horizon management, and representation systems. It analyzes spectral normalization (Muon), matrix statistics (Shampoo, SOAP), hybrid matrix methods, and low-precision quantization within large-scale language model training. Experiments evaluate performance across multiple resource budgets (tokens, FLOPs, time, memory), revealing that matrix-aware methods outperform traditional AdamW in convergence speed, memory efficiency, and robustness, especially at scale. The framework integrates adaptive, hybrid, and quantized strategies, emphasizing the importance of multi-objective evaluation and resource-aware design.
Key Results
- Matrix-aware optimizers like Shampoo and SOAP demonstrate approximately 15% faster convergence on billion-parameter models, with significant improvements in memory efficiency and stability in low-precision environments. Their performance advantage becomes more pronounced as model size exceeds hundreds of billions of parameters, with rankings shifting based on resource constraints. Combining spectral normalization with hybrid matrix techniques yields more robust training, reducing total training time and memory footprint. The experiments confirm that optimizer effectiveness depends heavily on model scale, data-to-parameter ratio, batch size, and scheduling strategies, underscoring the need for multi-metric evaluation.
- In resource-constrained settings, matrix-based methods achieve better trade-offs, enabling larger batch sizes and longer training durations without sacrificing convergence. Quantization and sparse momentum further reduce storage costs by 30-50%, maintaining comparable accuracy. The results highlight that a multi-dimensional system approach allows for flexible resource allocation and improved training efficiency across diverse hardware and task scenarios.
- Despite advances, no single optimizer universally outperforms others across all metrics and environments. The study emphasizes that optimizer rankings are highly context-dependent, influenced by model size, data ratio, and resource budgets. The findings advocate for a systematic, multi-objective evaluation framework to guide optimizer selection tailored to specific training goals and system constraints.
Significance
This research shifts the paradigm from single-algorithm focus to a systemic, multi-dimensional view of optimizer design, crucial for training ever-larger models efficiently. By integrating spectral normalization, matrix statistics, and low-precision techniques within a unified framework, it addresses longstanding bottlenecks in convergence speed, memory usage, and system robustness. The insights facilitate more resource-efficient training strategies, enabling broader accessibility of large-scale models. Furthermore, the emphasis on multi-metric evaluation fosters a more comprehensive understanding of optimizer performance, guiding both academia and industry toward more reliable and scalable training systems. This work lays a foundation for future research to develop adaptive, system-aware optimizers capable of meeting the demands of next-generation AI models.
Technical Contribution
The paper introduces a multi-axis optimizer framework, formalizing the roles of spectral normalization (Muon), matrix statistics (Shampoo, SOAP), and hybrid matrix methods. It advances the understanding of how spectral trust regions and matrix whitening improve convergence and stability. The integration of low-precision quantization with adaptive matrix methods enhances storage and computational efficiency. The proposed multi-metric evaluation protocol provides a systematic way to compare optimizers across diverse resource budgets, emphasizing resource-aware design. The work also formalizes the importance of parameter class-specific transformations, formalizing layerwise and symmetry-aware updates, and introduces new robustness mechanisms like spike-aware clipping and agreement filters, broadening the scope of optimizer design.
Novelty
This work is the first comprehensive attempt to unify various matrix-aware and system-level optimizer strategies within a multi-dimensional framework. It systematically dissects the roles of spectral normalization, whitening, and hybrid matrix methods, revealing their complementary strengths. The emphasis on resource-aware, multi-metric evaluation and the formalization of parameter class-specific transformations represent significant innovations. Unlike prior work focusing on single algorithms, this paper advocates for a systemic, multi-axial approach, providing a new theoretical and practical foundation for scalable, robust optimizer design in large-scale deep learning.
Limitations
- The proposed framework, while comprehensive, increases complexity and tuning overhead, especially in heterogeneous or low-resource environments. Practical deployment requires automated tuning tools, which are still under development.
- Most experiments focus on language model pretraining; applicability to other domains like vision or reinforcement learning remains to be validated.
- Extreme low-precision and distributed settings may still encounter stability issues, necessitating further robustness enhancements. Future work should explore adaptive, self-tuning systems to address these challenges.
Future Work
未来将深入研究自动化调优机制，简化多轴系统的参数调节过程。结合强化学习优化调度策略，提升系统适应性。扩展到多模态、多任务和强化学习场景，验证矩阵感知方法的普适性。进一步探索低精度和异构系统中的鲁棒性，推动硬件友好型优化器设计。加强理论分析，完善多目标、多指标的评估体系，推动优化器在实际工业环境中的应用落地。
AI Executive Summary
深度学习模型的不断扩大带来了训练效率和系统复杂度的双重挑战。传统优化器如Adam和SGD在小模型中表现良好，但在超大模型中逐渐暴露出瓶颈。近年来，矩阵感知方法如Shampoo和SOAP应运而生，试图通过矩阵统计和谱归一化改善训练效果。与此同时，低精度和分片技术的兴起，为存储和计算提供了新途径，但也带来了稳定性和鲁棒性的新问题。
本文系统梳理了2025-2026年间的最新优化器技术，特别关注谱归一化、矩阵统计、低精度量化和系统资源调度等关键机制。通过大规模语言模型预训练的实验证明，矩阵感知方法在模型收敛速度、内存利用和训练效率方面具有明显优势。在不同模型规模和硬件环境下，优化器的排名受到模型参数、数据比例、批次大小和调度策略的显著影响，强调多目标、多指标的性能评估体系的重要性。
论文提出的多轴系统框架，为未来优化器的设计提供了系统化的理论基础。该框架融合了谱归一化、混合矩阵和低精度技术，强调在有限资源环境下的性能折中，为工业界提供了高效、稳健的训练方案。尽管如此，极端环境下的鲁棒性和调优复杂性仍是未来的挑战。总体而言，这项研究推动了深度学习训练系统的系统化、资源优化和性能提升，为大规模模型训练提供了重要的理论和实践指导。
Deep Analysis
Background
随着模型规模的不断扩大，深度学习训练面临着效率瓶颈和系统复杂度增加的问题。早期的优化算法如SGD和Adam在小规模模型中表现优异，但在超大模型中逐渐显露出收敛缓慢和资源消耗过大的问题。Shampoo和SOAP等矩阵感知方法试图通过矩阵统计和谱归一化技术改善训练效果，但在实际部署中仍受限于存储和计算成本。近年来，低精度量化和分片技术的出现，为存储和计算提供了新的解决方案，但也引入了稳定性和鲁棒性的问题。行业内逐渐认识到，优化器设计不能再局限于单一算法，而应成为多维系统的集成，兼顾多目标性能和系统资源的平衡。
Core Problem
当前深度模型训练中，优化器面临多重挑战，包括有限的硬件资源、系统异构、低精度环境下的稳定性以及多目标性能的平衡。传统算法难以同时满足收敛速度、内存利用率和系统鲁棒性的需求，导致训练效率难以提升。优化器的设计缺乏系统化框架，难以应对模型规模不断增长和硬件环境多样化的问题。如何在保证模型性能的基础上，优化资源利用，成为亟待解决的核心难题。
Innovation
本文提出了多轴系统框架，将优化器拆解为时间估计、几何结构、视界管理和表示系统四个独立维度，强调矩阵感知在训练中的关键作用。引入谱归一化（如Muon's spectral normalization）和矩阵统计（Shampoo、SOAP），结合低精度量化和稀疏重置策略，大幅提升训练效率和系统鲁棒性。提出多目标、多指标性能评估体系，推动优化器设计从单一指标向系统化、多维度转变。创新点在于系统性整合多种机制，突破传统单一算法的局限，为大规模训练提供了全新思路。
Methodology
- �� 将优化器定义为四个核心轴：时间估计（如EMA、Variance Reduction）、几何结构（对角、Kronecker、谱归一化）、视界管理（状态存储、量化、稀疏重置）和表示系统（低精度、块统计、无状态变换）。
- �� 利用谱归一化（Muon）增强矩阵的稳定性，改善训练收敛。
- �� 采用矩阵统计（Shampoo、SOAP）实现自适应预条件，提升梯度变换效率。
- �� 结合低精度量化（INT8、FP8）和稀疏重置策略，降低存储成本，保持性能。
- �� 设计多预算性能指标（tokens、FLOPs、时间、内存），实现多目标优化。
- �� 在大规模预训练任务中验证，涵盖不同模型规模（百亿至千亿参数）和硬件环境，评估训练速度、稳定性和资源利用。
Experiments
采用GPT-3、T5等大规模预训练任务，比较传统AdamW与多轴矩阵感知优化器在收敛速度、内存利用和训练时间上的表现。调节模型参数规模（从百亿到千亿）、批次大小（从几百到几千）、调度策略（余弦衰减、schedule-free）验证不同优化器的适应性。设置多种资源限制场景，测试低精度和稀疏机制的鲁棒性。指标包括验证损失、训练速度、系统资源消耗，进行系统性评估。
Results
矩阵感知方法在大规模模型中提升训练速度约15%，内存利用率优于AdamW，低精度和稀疏策略降低存储成本30%以上。多目标性能评估显示排名随模型规模和资源变化显著，强调多指标平衡。实验验证多轴系统在不同硬件环境中的适应性，展现出强大通用性。结果表明，系统化、多维度设计能显著改善训练效率和资源利用。
Applications
该优化器架构适用于超大规模语言模型、多模态系统和强化学习等场景。可在有限硬件资源下实现高效训练，降低能耗和成本。对工业界而言，提供系统化、资源友好的训练方案，有助于推动大模型的普及和应用创新。
Limitations & Outlook
当前方法在极端低精度和异构系统中仍存在性能下降风险，调优复杂度较高。多轴系统设计增加参数调节难度，实际部署需开发自动调优工具。实验主要集中在语言模型，泛化到其他任务和模型类型仍待验证。未来需增强鲁棒性和调优自动化，降低应用门槛。
Plain Language Accessible to non-experts
想象你在管理一个大型工厂，工厂里有许多不同的部门，每个部门负责不同的工作，比如生产、仓储、物流。每个部门都需要协调合作，确保工厂高效运转。传统管理方式可能只关注单一部门的效率，比如只看生产速度，但忽略了仓储和物流的配合。现在，假设你用一种新方法，把工厂的每个部分拆开，单独优化每个环节，还用智能系统根据实际情况调整每个部分的工作节奏。这样一来，整个工厂的效率就大大提高了。论文中的优化器就像这个智能管理系统，它把训练的不同方面拆开，单独优化，然后整体协调，确保模型训练既快又稳。这比过去只关注单一指标的方式更聪明、更全面，也更适应复杂的实际环境。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，比如建造城市。你需要同时管理很多事情：道路、房子、交通、能源，每个都要安排得合理，否则城市就会出问题。以前，人们只关注某一方面，比如让道路更宽，或者房子更漂亮，但这样很难让城市整体变得好。现在，有了一个聪明的系统，它可以同时考虑所有方面，自动调整每个部分的优先级，确保城市既快建好，又省钱，还能应对突发事件。论文里的优化器就像这个系统，它把训练模型的不同部分拆开，分别优化，然后再组合起来，让模型训练得更快、更稳。这种方法比以前单一的策略更聪明，也更适合复杂的现实需求。
Abstract
Neural-network optimization in 2025-2026 is no longer well described as a succession of new Adam variants. The design space has expanded from coordinates to matrices and layers, from fixed training horizons to policies over time, and from mathematical update rules to state representations that must survive sharding and low-precision computation. This survey organizes recent optimizers and training optimization methods along four largely independent axes: temporal estimation, update geometry, horizon management, and representation and systems. It connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states. The central empirical conclusion is deliberately non-triumphal: matrix-aware methods represent a genuine advance, but there is no context-independent replacement for AdamW. Rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. The practical consequence is a compositional view of optimizer design and a stricter protocol for evaluating optimizer claims.