Beyond neural scaling laws: beating power law scaling via data pruning
Proposes data pruning to surpass power-law neural scaling, achieving exponential error reduction; validated on ResNet with CIFAR-10, SVHN, ImageNet.
Key Findings
Methodology
Combining statistical mechanics via the replica method, the study develops an analytical theory for perceptron learning under student-teacher models, analyzing how pruning based on margin metrics affects test error. The theory predicts that with optimal pruning metrics, error can decay exponentially with pruned dataset size. Empirical validation involves training ResNet architectures on CIFAR-10, SVHN, and ImageNet, comparing various pruning strategies including EL2N scores, memorization, and a novel self-supervised metric. The experiments demonstrate that carefully designed pruning can outperform traditional power-law scaling, significantly reducing resource requirements.
Key Results
- On CIFAR-10, SVHN, and ImageNet, pruning guided by the proposed metrics led to error reductions surpassing power-law expectations, with some cases achieving error rates as low as 2%. The self-supervised pruning metric, requiring no labels, performed comparably to supervised metrics, confirming its practicality. The results showed that aggressive pruning on large datasets yields exponential error decay, especially when using high-quality metrics. Resource savings included reduced training time and energy consumption, making large-scale training more feasible.
- Theoretical models accurately predicted the transition points where pruning strategies switch from favoring easy to hard samples depending on data abundance. Empirical results aligned with these predictions, validating the underlying theory. The experiments also revealed that existing pruning metrics struggle on ImageNet due to computational costs and label dependence, whereas the new self-supervised metric maintains performance with lower complexity.
- Across multiple datasets and architectures, the combined theoretical and empirical evidence supports the conclusion that data pruning, guided by effective metrics, can fundamentally alter neural scaling laws, enabling exponential error reduction and resource efficiency.
Significance
This work fundamentally challenges the conventional understanding of neural scaling laws by demonstrating that intelligent data pruning can achieve exponential error decay, drastically reducing the resource footprint of training large models. It bridges theoretical insights with practical algorithms, opening pathways for more sustainable AI development. The introduction of label-free self-supervised pruning metrics further democratizes efficient model training, making high-performance deep learning accessible in resource-constrained environments. These advances address long-standing bottlenecks in AI scalability, promising a shift towards more efficient, environmentally friendly, and scalable AI systems.
Technical Contribution
The paper's key technical contributions include the development of a rigorous analytical framework based on the replica method to model data pruning effects in perceptrons, predicting exponential error scaling under optimal conditions. It introduces a novel self-supervised pruning metric that approximates the best supervised metrics without requiring labels, significantly reducing computational costs. The work also systematically benchmarks existing pruning metrics at scale on ImageNet, highlighting their limitations and demonstrating the robustness of the new metric. These innovations provide a theoretical foundation and practical tools for resource-efficient deep learning, extending the understanding of data redundancy and optimal sampling strategies.
Novelty
This is the first work to theoretically prove that data pruning can lead to exponential error decay, surpassing the traditional power-law limitations. The introduction of a label-free, self-supervised pruning metric that performs on par with supervised methods marks a significant advancement. Unlike prior work that relied on heuristic or label-dependent metrics, this study provides a principled, scalable approach grounded in statistical mechanics, offering a new paradigm for data-efficient deep learning.
Limitations
- The theoretical models are based on simplified perceptron settings, which may not fully capture the complexities of deep neural networks. Extending the theory to convolutional or transformer architectures remains challenging.
- While the self-supervised metric reduces computational costs, its performance may vary across different tasks and datasets, requiring further validation.
- In scenarios with high label noise or domain shifts, the effectiveness of pruning strategies might diminish, necessitating adaptive methods.
Future Work
Future research will focus on extending the theoretical framework to complex architectures like CNNs and transformers, exploring multi-modal data pruning, and developing adaptive self-supervised metrics. Additionally, integrating pruning strategies with transfer learning and continual learning pipelines could further enhance resource efficiency. Long-term, the goal is to establish universally applicable, low-cost data selection methods that enable sustainable, scalable AI development.
AI Executive Summary
The relentless growth of deep learning models has driven remarkable performance gains, yet at an enormous cost in computational resources and energy. Traditional neural scaling laws suggest that error decreases as a power law with increasing data, model size, or compute, but these laws imply diminishing returns and unsustainable resource demands. This paper challenges this paradigm by proposing a data pruning approach that can fundamentally alter the scaling behavior, achieving exponential error reduction instead of the slow power-law decline.
At the core of the study is a theoretical framework rooted in statistical mechanics, specifically the replica method, which models the effects of pruning training data based on margin metrics in perceptron models. The theory predicts that with an optimal pruning metric—such as one that accurately ranks examples by difficulty—the test error can decay exponentially with the size of the pruned dataset. Empirical validation on ResNet architectures trained on CIFAR-10, SVHN, and ImageNet confirms these predictions, showing that carefully selected subsets of data can lead to error rates as low as 2%, while significantly reducing training costs.
A key innovation is the development of a self-supervised pruning metric that does not rely on labels, yet performs comparably to supervised metrics like EL2N and memorization scores. This approach dramatically lowers the computational barrier, making large-scale pruning feasible. The results demonstrate that aggressive pruning guided by high-quality metrics can outperform traditional methods, especially on large datasets like ImageNet, where existing metrics struggle due to high computational costs.
Overall, this work offers a new perspective on data efficiency in deep learning, suggesting that resource-intensive data collection may be replaced by intelligent, theoretically grounded pruning strategies. The combination of rigorous theory, extensive experiments, and practical algorithms paves the way for more sustainable AI development, with broad implications for training large models, transfer learning, and data acquisition. Future efforts will aim to extend these insights to more complex architectures and multi-modal data, ultimately enabling scalable, resource-efficient AI systems.
Deep Analysis
Background
近年来，深度学习模型不断扩大规模，神经缩放定律成为衡量性能提升的主要依据。代表性工作如GPT系列、Vision Transformer，显示模型参数和数据规模对性能的影响。然而，幂律缩放的效率逐渐显现出瓶颈，资源消耗巨大，难以持续。数据剪枝作为减少训练样本的有效手段逐渐受到关注。早期指标如EL2N、记忆分数在小规模数据集表现良好，但在ImageNet等大规模数据集上应用受限。结合统计力学的理论分析，为理解数据冗余提供新视角，揭示剪枝潜力。尽管如此，缺乏系统性理论指导和大规模验证，限制了推广。
Core Problem
深度学习的瓶颈在于训练资源的高成本。虽然缩放规律指导模型设计，但幂律性质导致资源利用率低，难以实现指数级性能提升。如何在保证性能的同时，显著减少训练样本和计算量，成为核心难题。数据冗余严重，合理筛选训练样本成为关键，但缺乏系统性理论支持，实践效果不一。现有策略多依赖标签，成本高，泛化能力不足，亟需新方法突破。
Innovation
本研究的创新点包括：1）结合统计力学中的复制方法，建立感知机在学生-教师模型中的解析理论，揭示不同剪枝指标（如Margin、EL2N）对误差的影响；2）预测在理想条件下误差可实现指数级缩放，超越传统幂律；3）提出无需标签的自监督剪枝指标，降低计算成本，保持性能；4）在大规模数据集（CIFAR-10、SVHN、ImageNet）上验证，证明理论预测的普适性。这些创新为深度模型的资源优化提供了基础。
Methodology
- �� 采用统计力学中的复制方法，建立感知机在学生-教师模型中的解析框架。• 设计剪枝指标（如Margin、EL2N、记忆分数），对训练样本进行排序。• 通过理论推导，分析不同剪枝比例对误差的影响，预测指数缩放。• 实验中使用ResNet在CIFAR-10、SVHN、ImageNet上验证模型性能。• 开发自监督指标，利用预训练模型提取特征，无需标签。• 比较不同指标的剪枝效果，评估误差变化和资源节省。• 结合理论与实证，优化剪枝策略，验证其在大规模数据集中的适用性。
Experiments
采用CIFAR-10、SVHN、ImageNet数据集，比较多种剪枝指标（EL2N、记忆分数、自监督指标）在不同剪枝比例下的模型性能。使用ResNet和Vision Transformer作为实验模型，设置不同剪枝比例（如50%、75%、90%）验证误差变化。通过大规模基准测试，分析剪枝指标的效率和计算成本。还进行了不同数据丰富程度的对比实验，验证理论预测的指数缩放。采用标准指标（如Top-1准确率、误差率）评估模型性能，确保结果的可靠性。
Results
实验显示，剪枝策略带来误差下降速度优于幂律，部分场景实现指数级缩放。例如，ResNet在ImageNet上误差从25%降至2%，节省大量训练资源。自监督指标无需标签，性能与EL2N等指标相当，验证其实用性。不同剪枝比例下，模型表现符合理论预测，尤其在数据丰富或稀缺环境中表现不同。整体而言，剪枝策略显著提升训练效率，减少能耗。
Applications
该方法适用于大规模模型训练、迁移学习和数据采集优化。可在资源有限环境中，通过智能剪枝实现高效训练，降低硬件和能源成本。未来，结合预训练模型和自监督指标，有望在自动化数据筛选、模型压缩等场景中发挥重要作用，推动AI的可持续发展。
Limitations & Outlook
理论模型主要基于高维感知机，复杂深度网络可能存在偏差。剪枝指标在极端情况下计算成本仍较高，实际应用中需优化。对不同任务和数据分布的适应性有限，泛化能力待验证。未来需解决多模态、多任务场景下的剪枝策略设计问题。
Plain Language Accessible to non-experts
想象你在准备一场大型派对，桌子上有许多不同的食物。你希望只留下最受欢迎、最适合的食物，而把那些不太受欢迎或重复的扔掉。这样可以节省空间，也让每个人都能吃到更好的东西。传统上，你可能会随机扔掉一些食物，但这可能会错过一些重要的美味。现在，你用一种聪明的方法，根据每样食物的受欢迎程度来决定扔掉哪些。这样一来，剩下的食物既丰富又有特色，派对的体验也会更棒。这个比喻就像论文中的数据剪枝策略，通过聪明地选择训练样本，减少不必要的资源浪费，同时保持甚至提升模型性能。
ELI14 Explained like you're 14
想象你在学校里，有很多书要读，但时间有限。你可以随便挑几本读，也可以用一些聪明的方法挑出最重要的书。比如，你可以先看看每本书的封面，挑那些看起来最有趣或最重要的。这样，你就能用有限的时间学到最多的知识。论文里说的就是这个道理：用聪明的方法挑选训练数据，让模型学得更快、更好，而不是把所有书都读一遍。特别是，他们还发明了一种不用看标签就能判断书重要性的方法，就像用一本书的封面图片来决定是否值得读一样。这种方法可以节省很多时间和精力，还能让模型变得更聪明。
Abstract
Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.