Muon Learns More Robust and Transferable Features than Adam
Muon optimizer outperforms Adam in robustness and transferability, with larger logit margins and higher spectral rank in features.
Key Findings
Methodology
This study evaluates pretrained models' robustness via corrupted datasets (ImageNet-C, FineWeb-C) and analyzes hidden states using layer-wise probes. Spectral analysis of hidden representations quantifies feature richness. Theoretical models demonstrate Muon's spectral regularization enhances feature margins and rank, explaining empirical advantages. Experiments compare Muon, Adam, and SGD across vision and language tasks, measuring accuracy, perplexity, and spectral properties to establish robustness and transferability correlations.
Key Results
- Muon models achieve highest mean accuracy on corrupted datasets: ResNet-18 +3.2%, ViT-S +2.8%, GPT-2 perplexity reduced by over 15%. Layer probes show margins 20% larger across layers, indicating stronger class separation. Spectral rank analysis reveals Muon features span richer subspaces, facilitating transfer.
- In downstream tasks, Muon-pretrained models outperform Adam, with higher effective rank and better adaptation performance. The spectral normalization in Muon reduces singular-value imbalance, leading to more robust and transferable features.
- Theoretical analysis confirms Muon’s spectral regularization removes gradient spectral imbalance, provably yielding larger margins and higher feature rank than Adam and gradient descent, underpinning empirical findings.
Significance
This work advances understanding of how optimizer choices influence feature quality, not just training speed. Demonstrating Muon’s ability to produce more robust and transferable features addresses key challenges in deploying large models in real-world scenarios. It bridges optimization mechanics with feature representation theory, offering new directions for designing algorithms that enhance model generalization, robustness, and adaptability across domains. Such insights are crucial for scaling trustworthy AI systems.
Technical Contribution
The paper introduces a spectral normalization perspective on Muon, linking its matrix orthogonalization to improved feature margins and spectral richness. It combines empirical evaluations with theoretical proofs, establishing that spectral regularization in Muon leads to larger class margins and higher effective rank in hidden states. The methodology includes layer-wise probing, spectral analysis, and simplified models to elucidate the mechanisms behind Muon’s advantages, providing a comprehensive framework for future optimizer development.
Novelty
This is the first comprehensive study connecting Muon’s matrix-based spectral regularization to concrete feature quality improvements. Unlike prior work focusing solely on optimization speed, this research quantifies how Muon enhances feature robustness and transferability through spectral properties. The theoretical model explicitly demonstrates the spectral balance effect, offering a novel explanation for Muon’s empirical success and establishing a new paradigm in optimizer-feature relationship analysis.
Limitations
- Current experiments are limited to vision and language tasks; broader modalities and real-world scenarios need validation.
- Theoretical models are simplified, assuming linearity and idealized spectral conditions, which may not fully capture deep network complexities.
- Computational overhead of spectral normalization may limit large-scale deployment without further optimization.
Future Work
Future research will extend Muon’s application to multi-modal and reinforcement learning tasks, exploring spectral dynamics during training. Developing more efficient spectral regularization techniques and integrating sparsity could improve scalability. Deeper theoretical models will aim to better approximate real network behaviors, and empirical studies will validate robustness in diverse, real-world environments. Additionally, combining Muon with other regularization strategies may unlock further gains in feature quality.
AI Executive Summary
In recent years, the quest for more robust and transferable features in deep neural networks has become central to advancing AI capabilities. While traditional optimizers like Adam and SGD have driven progress, their limitations in feature quality remain. Muon, a matrix orthogonalization optimizer, has shown remarkable efficiency in training large models, but its impact on feature learning was less understood. This study provides a comprehensive analysis, revealing that Muon not only accelerates training but also enhances the quality of learned features.
Through extensive experiments on vision and language tasks, Muon-pretrained models demonstrate superior robustness against input corruptions, achieving higher accuracy and lower perplexity than Adam and SGD. Layer-wise probing indicates that Muon produces larger logit margins across all layers, signifying stronger class separation and stability. Spectral analysis of hidden states shows that Muon models span richer feature subspaces, with higher effective ranks, facilitating better transfer to downstream tasks.
Theoretical models further clarify these empirical observations, illustrating how spectral regularization in Muon balances gradient spectra, leading to improved margins and feature diversity. These insights bridge the gap between optimization mechanics and feature representation, offering a new perspective on how optimizer design influences model generalization. The findings suggest that spectral normalization mechanisms can be key to developing future optimizers that promote robust, transferable features.
Looking ahead, expanding Muon’s application scope, optimizing its computational efficiency, and deepening theoretical understanding will be vital. This work paves the way for more reliable and adaptable AI systems, addressing critical challenges in deploying large-scale models in real-world environments. Overall, Muon’s dual benefits of speed and feature quality mark a significant step forward in deep learning research and practice.
Deep Analysis
Background
深度学习模型的快速发展带来了模型规模的指数增长，但训练效率和模型泛化能力仍是核心难题。Adam作为主流优化器，因其自适应特性被广泛采用，但在鲁棒性和迁移性方面仍有局限。近年来，Muon作为一种矩阵正交化优化器崭露头角，显示出在训练速度和模型性能上的潜力。相关研究主要关注其在大模型中的表现，但对其特征学习能力的系统性分析尚不足。理解Muon如何影响模型的特征空间结构，有助于推动优化器设计和模型泛化能力提升。
Core Problem
核心问题在于，尽管Muon在训练效率方面优于Adam和SGD，但其在特征质量上的优势尚未被充分揭示。具体表现为模型在输入扰动下的鲁棒性和迁移能力。传统指标如训练损失和准确率无法全面反映特征的判别性和丰富性，因此需要引入边界和谱秩等指标进行深入分析。如何量化Muon在多层次特征空间中的优势，成为研究的关键难题。
Innovation
本研究创新点包括：1）提出基于logit边界和谱秩的特征质量评估体系，系统比较Muon、Adam和SGD；2）结合层级探针和谱分析，揭示Muon在多层次特征空间中的机制优势；3）建立简化模型，理论证明Muon通过谱正则化提升边界和特征秩，优于传统优化器。这些创新为理解深度模型的特征学习提供了新视角，也为优化器设计提供理论基础。
Methodology
- �� 输入：预训练模型（ResNet-18、ViT-S、GPT-2）和腐蚀数据集（ImageNet-C、FineWeb-C）。
- �� 训练：采用标准化训练方案，确保公平比较。
- �� 鲁棒性评估：在腐蚀数据上测试模型性能，记录准确率和困惑度。
- �� 特征分析：用层级探针训练线性解码器，计算每层的logit边界。
- �� 表示空间分析：对隐藏状态进行奇异值分解，计算谱秩。
- �� 理论验证：建立简化模型，分析谱正则化对特征边界和秩的影响。
Experiments
在ImageNet-C和FineWeb-C上测试预训练模型鲁棒性，比较Muon、Adam和SGD的性能差异。采用Top-1准确率和困惑度作为指标，调优超参数确保公平。层级探针用于分析每层的logit边界，谱分析评估隐藏状态的特征丰富性。迁移实验中，训练线性分类器和微调模型，验证特征迁移能力。理论分析通过简化模型证明Muon的谱正则化优势。
Results
Muon在腐蚀数据上表现优异，ResNet-18平均准确率提升3.2%，ViT-S提升2.8%，GPT-2模型困惑度降低15%以上。层级探针显示Muon模型各层logit边界显著大于Adam和SGD，平均提升20%以上。谱分析表明Muon模型隐藏状态的有效秩更高，表示空间更丰富，迁移性能更优。这些结果验证了Muon在特征判别性和鲁棒性上的优势。
Applications
Muon优化器可广泛应用于大规模预训练模型，提升模型在实际场景中的鲁棒性和迁移能力。适用于图像识别、自然语言处理等领域，尤其在对抗攻击和域适应任务中表现优越。未来结合稀疏化和分布式训练，将进一步推动其工业应用。长远来看，Muon有望成为深度学习模型的标准优化工具，助力智能系统的稳健发展。
Limitations & Outlook
目前研究主要集中在图像和文本任务，尚未验证多模态或复杂场景的表现。理论模型简化，假设线性和理想谱条件，可能无法完全反映深层网络的复杂交互。计算成本略高，实际部署需优化算法效率。未来需扩展多任务、多模态应用，提升实用性。
Plain Language Accessible to non-experts
想象你在一个工厂里生产各种产品。每个工序都需要不同的工具和方法。有些工具能快速完成任务，但可能不够稳妥；而有些工具虽然慢点，但能保证产品质量。Muon就像一种新型的工具，它不仅能更快地完成工作，还能确保生产出的产品在遇到意外情况时还能保持质量。它通过特殊的“调节”方式，使得每个工序都能更稳健、更高效。这样，工厂的整体表现就更好，不容易出错，也更容易把产品转移到其他工厂继续生产。这个“工具”在训练深度神经网络时起到了类似作用，让模型更强大、更可靠。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。每次拼完一块，都会影响下一块的拼法。有些拼图方法虽然快，但拼出来的图像容易变形或不稳。Muon就像一种特别聪明的拼图策略，它不仅拼得快，还能确保拼出来的图像非常稳固，遇到风吹雨打也不散。它会用一种特别的“平衡术”调整每一块拼图的位置，让整体更稳固、更漂亮。而且，这种拼图方法还能把拼好的图像轻松搬到别的房间继续用，不会丢失细节。这样，拼图变得既快又稳，效果也更好。深度学习里的模型就像这个拼图游戏，Muon帮模型拼出更稳固、更聪明的“图像”。
Abstract
Muon has recently emerged as a state-of-the-art optimizer for pretraining Large Language Models (LLMs) and vision classifiers. Despite its efficiency advantage over Adam and SGD, the feature-learning advantage of Muon remains unclear. This paper investigates Muon's feature-learning advantage through the lens of robustness and transferability. First, by evaluating pretrained models on corrupted images and texts, we show that features learned by Muon are consistently more robust than those learned by Adam and SGD across different architectures, including transformers and Convolutional Neural Networks (CNNs). Using trained layer-wise probes, we further show that this robustness advantage is reflected in larger logit margins across layers. Second, by training linear classifiers or fine-tuning full models from pretrained parameters on downstream tasks, we demonstrate that Muon-learned features transfer more effectively than those learned by Adam and SGD. This transferability advantage is further supported by the diversity of hidden states across layers, as measured by effective rank. Finally, in a representative classification problem with multi-component features, we prove that Muon attains larger margins and higher effective rank than Adam and SGD, providing theoretical support for our empirical findings.