Are Sparse Neural Networks Better Hard Sample Learners?

The evolution of deep learning has led to increasingly large models like ResNet and Transformer, achieving state-of-the-art results but at the expense of massive computational resources. Sparse neural networks (SNNs) emerged as a solution to reduce model size and inference cost through pruning and sparse training techniques such as GMP, LTH, SNIP, and SET. While these methods excel on standard datasets, their performance on complex, noisy, or adversarial samples remains underexplored. As AI applications expand to edge devices and safety-critical systems, understanding how sparsity affects learning from hard samples becomes crucial. Prior work has shown sparse models can mitigate overfitting and improve robustness, but a systematic evaluation under challenging data conditions is lacking.

Hard samples—characterized by noise, complexity, or adversarial perturbations—pose significant challenges for deep neural networks, often leading to overfitting and poor generalization. Existing research primarily focuses on dense models, leaving a gap in understanding sparse models' capabilities in these scenarios. The core issue is whether sparse networks, which eliminate redundant weights, can effectively learn from such difficult data, especially under limited data regimes. Additionally, how layer-wise density distribution influences performance remains unclear, complicating the design of efficient sparse models for real-world tasks.

The paper introduces a comprehensive framework combining EL2N-based difficulty assessment with multiple sparsification algorithms to evaluate performance on challenging samples. It emphasizes the importance of layer-wise density ratios, particularly in shallow layers, for improving sparse model learning. The study uniquely investigates the impact of training data volume, external perturbations, and sparsity strategies, revealing that sparse models can outperform dense ones in robustness and accuracy under complex conditions. This approach bridges the gap between model compression and hard sample learning, offering new insights into sparse network design.

• �� Use EL2N scores to quantify sample difficulty, selecting top 50% hardest samples for training.• Implement multiple sparsification strategies (GMP, LTH, SNIP, SET) across different sparsity levels (10%-90%).• Train models on CIFAR-100 and TinyImageNet using ResNet architectures, comparing from scratch and pre-trained initializations.• Evaluate performance under various challenging conditions: intrinsic complexity, external noise, adversarial attacks.• Analyze layer-wise density ratios, focusing on shallow vs. deep layer configurations.• Conduct ablation studies on different sparsity distributions, including ERK, uniform, and random pruning, to assess their impact on robustness and accuracy.

Experiments involved training ResNet18 and ResNet34 on CIFAR-100 and TinyImageNet, with sparsity ratios from 10% to 90%. The models were tested on clean, noisy, and adversarially perturbed data, including high EL2N samples and corrupted images. Performance metrics included accuracy, robustness to attacks, and computational cost (FLOPs). Multiple sparsification methods were compared, emphasizing layer-wise density distribution. Experiments also examined data volume effects, with reduced datasets (20%-30%) highlighting the robustness of sparse models under limited data. Results were validated through repeated trials and ablation studies on density configurations.

Sparse models at moderate sparsity (10%-80%) consistently matched or exceeded dense model accuracy, especially on hard samples and limited data. SET and SNIP outperformed dense baselines by 3-5% at high sparsity, with enhanced robustness against adversarial attacks and noise. Layer-wise density analysis revealed that higher shallow-layer density correlates with better performance, especially when training from scratch. Under limited data (20%-30%), sparse models achieved 2-4% higher accuracy, demonstrating their resilience and efficiency. These findings confirm that strategic sparsity enhances learning from complex, noisy, and adversarial data.

The findings support deploying sparse models in edge devices, autonomous systems, and security applications where computational resources are limited but robustness is critical. Efficient sparse training can reduce energy consumption and inference latency while maintaining high accuracy on challenging tasks. Additionally, the insights into layer-wise density tuning can inform hardware-aware model design, enabling more effective deployment of AI in real-world, resource-constrained environments.

The study primarily focuses on image classification; applicability to other domains like NLP or object detection remains to be validated. Extreme sparsity (>90%) results in performance degradation, indicating the need for more advanced sparsification algorithms. Most methods are unstructured sparse, which may not be hardware-efficient. Further research is needed to develop structured sparsity techniques and explore real-time dynamic pruning for broader practical deployment.