Asymmetric Scaling Laws from Sparse Features
Proposes a sparse activation-based neural scaling law model, revealing double descent peaks and asymmetry due to sparsity effects.
Key Findings
Methodology
This paper constructs a sparse activation random feature model analyzing asymptotic training loss across parameter and data scales. By deriving limit expressions for the loss in under- and overparameterized regimes, it uncovers a double-exponent scaling law driven by sparsity. The model combines high-dimensional random embeddings with heavy-tailed sparse distributions, employing tools from asymptotic analysis and random matrix theory to validate the existence of a double descent peak near the interpolation threshold. Additionally, the study examines gradient descent stability probabilities and the persistence of sparsity effects under nonlinear activations.
Key Results
- The model exhibits two distinct decay exponents: αN=α1+α2+1 in the underparameterized regime and αD=(α1+α2+1)/(α1+1) in the overparameterized regime, with a pronounced double descent peak near the interpolation point.
- Under fixed compute budgets, the analysis shows that increasing dataset size yields more efficient error reduction than enlarging model capacity, establishing a compute-optimal frontier favoring data expansion.
- Numerical experiments confirm the theoretical exponents and the double descent phenomenon, demonstrating that sparsity-induced asymmetry persists in nonlinear two-layer networks, indicating broad applicability.
Significance
This work pioneers a systematic theoretical framework illustrating how sparsity fundamentally alters neural scaling laws. By revealing the intrinsic asymmetry and double descent peaks caused by sparse features, it advances understanding of training efficiency and resource allocation in large models, especially in natural language processing and high-dimensional data contexts. The insights facilitate designing more resource-effective training strategies, enabling better performance with limited computational budgets. The findings bridge the gap between empirical scaling observations and theoretical explanations, offering a new lens to interpret the role of sparse features in deep learning.
Technical Contribution
The paper introduces a novel asymmetric two-exponent scaling law rooted in the structural properties of sparse high-dimensional data. It combines asymptotic limit analysis with random matrix techniques to derive explicit formulas for the loss in different regimes, revealing how sparsity induces a phase transition in scaling behavior. It also formulates a compute-optimal resource allocation strategy, emphasizing data-centric growth over model size under fixed budgets. The work extends classical random feature theory to incorporate sparsity effects, providing rigorous guarantees and broadening the scope of theoretical understanding in neural scaling laws.
Novelty
This is the first comprehensive analysis demonstrating that sparsity in feature activation causes intrinsic asymmetry in neural scaling laws, leading to distinct exponents in under- and overparameterized regimes. Unlike prior models assuming dense Gaussian inputs, this work explicitly models heavy-tailed sparse distributions, revealing phenomena like double descent peaks and resource allocation shifts driven by sparsity. It bridges theoretical insights with empirical validation, offering a new paradigm for understanding large-scale neural training with sparse data.
Limitations
- The model assumes idealized heavy-tailed Bernoulli activation distributions, which may not fully capture the complexity of real-world data sparsity patterns.
- Analysis focuses on asymptotic regimes, so finite-sample effects and practical deviations might limit direct applicability.
- Gradient dynamics are studied under fixed step sizes, not considering adaptive optimizers or more complex nonlinearities, which require further investigation.
Future Work
Future research should explore the impact of different sparsity structures, extend analysis to deeper nonlinear networks, and incorporate adaptive optimization algorithms. Empirical validation on real datasets with complex sparsity patterns, such as language models and recommendation systems, is also crucial. Additionally, integrating these insights into training protocols could lead to more efficient large-scale model development, especially under resource constraints.
AI Executive Summary
This study introduces a novel theoretical framework for understanding neural scaling laws under sparse feature activation. By constructing a sparse random feature model, the authors reveal that training loss exhibits a double descent phenomenon characterized by two distinct power-law exponents, αN and αD, governing the decay in underparameterized and overparameterized regimes respectively. The key insight is that sparsity induces a phase transition in the loss landscape, leading to an intrinsic asymmetry that was previously overlooked in dense models.
The analysis leverages asymptotic limit theorems and random matrix theory to derive explicit formulas for the population loss, demonstrating that the loss peaks near the interpolation threshold—a hallmark of the double descent phenomenon. The model predicts that in fixed compute scenarios, allocating resources toward increasing data rather than model size yields better performance, establishing a compute-optimal frontier that favors data expansion, especially as sparsity intensifies.
Empirical simulations validate the theoretical exponents and the double descent peak, with experiments extending to nonlinear two-layer networks confirming the persistence of asymmetry. These findings have profound implications for large-scale model training, suggesting that sparsity-aware strategies can significantly improve resource efficiency. Despite some limitations in modeling real-world data complexity, this work provides a foundational step toward understanding and exploiting sparsity in deep learning, opening avenues for more efficient, scalable AI systems in the future.
Deep Analysis
Background
近年来，神经网络的缩放规律成为理解模型性能提升的核心。Kaplan等提出的幂律关系揭示了模型参数和数据规模对测试误差的影响，Hoffmann等进一步提出了计算最优的缩放策略。随机特征模型和核方法为理论分析提供了基础，但大多假设输入为密集高斯分布，忽略了实际数据中的稀疏性。稀疏激活在自然语言和高维数据中普遍存在，尚未被充分理论化。现有研究多关注密集特征，缺乏对稀疏性引起的缩放不对称的系统理解。
Core Problem
核心问题在于，稀疏激活引起的特征分布偏离密集模型，导致训练误差在参数和数据规模下表现出不同的缩放指数。传统模型难以解释在稀疏特征条件下的双重下降峰和非对称规律。如何建立理论模型描述稀疏特征对训练误差的影响，揭示不同参数区间的极限行为，成为亟待解决的难题。这关系到大模型训练的效率优化和资源配置策略，具有重要的理论和实践价值。
Innovation
创新点包括：1) 提出稀疏激活引起的非对称缩放规律模型，区别于现有对称指数假设；2) 结合极限分析和随机矩阵理论，推导出欠参数和过参数区间的极限损失表达式；3) 发现稀疏性引发的双重下降峰，为理解大模型训练中的性能波动提供新视角；4) 提出固定计算预算下的最优资源配置策略，强调数据优先原则。这些创新丰富了神经网络缩放规律的理论体系，拓展了稀疏特征在深度学习中的应用边界。
Methodology
- �� 构建稀疏激活的随机特征模型，定义高维输入的稀疏分布和随机嵌入机制。
- �� 利用极限理论分析在不同参数规模下的训练误差，推导出极限损失表达式。
- �� 通过随机矩阵工具，分析稀疏特征引起的双重下降峰和缩放指数的变化。
- �� 计算固定预算下的最优模型和数据规模配置，推导出资源分配策略。
- �� 进行数值模拟验证理论预测，比较不同稀疏参数对缩放规律的影响。
Experiments
采用合成稀疏分布数据，模拟不同参数规模下的训练误差。通过调整稀疏指数α1、α2，验证双指数规律和双重下降峰的出现。使用两层非线性网络进行实证，观察稀疏激活引起的缩放不对称。实验中测量测试误差、模型参数和数据规模的关系，验证理论中的指数预测。还比较不同资源配置策略的效果，验证最优前沿的有效性。
Results
实验证明，损失在欠参数区间遵循指数αN=α1+α2+1，在过参数区间遵循αD=(α1+α2+1)/(α1+1)，且在参数规模相等时出现双重下降峰。资源配置分析显示，优先扩大数据集能显著降低误差，验证了理论中的最优策略。非线性网络实验也确认稀疏激活引起的缩放不对称持续存在，验证模型的广泛适用性。
Applications
该模型适用于自然语言处理、推荐系统等高维稀疏特征场景，指导大模型训练中的资源分配和模型设计。可用于优化预训练模型的规模选择，提升训练效率，减少计算成本。未来可结合实际数据分布，推广到深层网络和自适应优化器，推动稀疏特征在工业界的应用。
Limitations & Outlook
模型假设理想化稀疏分布，实际数据中的稀疏性更复杂，可能影响预测准确性。分析主要基于极限行为，未充分考虑有限样本和参数的偏差。梯度动态分析假设固定步长，未涉及自适应优化器和非线性激活的复杂动态，未来需扩展至更真实场景。
Plain Language Accessible to non-experts
想象你在一个工厂里工作，工厂里有很多机器，但只有少数几台在工作。这些机器代表数据中的重要特征，虽然数量少，但每台机器都非常关键，能带来巨大产出。工厂的管理者发现，专注于这些关键机器，可以用更少的资源做出更好的产品。这个故事类似于稀疏激活的神经网络：大部分特征其实没怎么用，但少数重要的特征却能决定整体表现。研究发现，当你只关注这些“重要的机器”时，模型的性能会表现出两种不同的成长规律：在参数少时，性能提升很快；而在参数多时，提升变得缓慢。更有趣的是，随着数据量的增加，模型的表现会在某个点出现“峰值”，之后反而变差，然后再次改善，就像工厂在调整机器数量时的波动。这些发现帮助我们理解如何在有限资源下，最大化模型的效果，特别在特征稀疏的情况下。
ELI14 Explained like you're 14
想象你在学校里，有很多不同的书包，但只有少数几本书在每次考试前会被用到。这些书代表那些特别重要的知识点，虽然平时很少用，但一旦用到就非常关键。老师发现，如果你只准备这些重要的书，你可以用更少的时间学更多的内容，效果还不错。可是，如果你准备得太少，很多重要的知识就学不到，成绩就会差。研究告诉我们，当你用很多书（模型参数）时，学习的效果会有两种不同的变化：在少书的阶段，学习很快变好；而在很多书的阶段，变得慢一些。更神奇的是，随着你学习的内容越来越多，成绩会出现一个奇怪的波动，先变差再变好，就像你在准备考试时的心情一样。这项研究帮助我们明白，怎么在有限的时间和资源里，最有效地学习那些最重要的知识点，特别是在信息很稀疏的情况下。
Abstract
We introduce a model for neural scaling laws under sparse activations. In the model, test loss is often dominated by rare coordinates that are never observed in the training input. This mechanism induces a novel bottleneck absent from dense models. We derive the asymptotic population loss in both the underparameterized and overparameterized regimes, and show that the loss exhibits a double-descent peak near the interpolation threshold -- where the number of parameters is just sufficient to fit the training data -- resulting in a loss curve governed by two distinct scaling exponents -- one for the overparameterized regime and one for the underparameterized regime -- with a gap determined by the degree of sparsity. Additionally, we derive a compute-optimal frontier that favors increasing dataset size over model capacity under fixed compute budgets. We also analyze gradient-descent dynamics and identify a scaling law for the probability that fixed-step gradient descent becomes unstable. We further show that the sparsity-induced effect persists under nonlinear activations.