Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing
This study demonstrates that fixing most parameters in deep CNNs and training only a small subset achieves near state-of-the-art performance.
Key Findings
Methodology
The authors fix nearly all weights in deep convolutional networks to random values, only training a small subset using SGD. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet across architectures like DenseNet, Wide-ResNet, VGG, and AlexNet evaluate the impact of partial parameter training. Techniques include layer-wise slicing, filter subset selection, and BN parameter tuning. The study controls training epochs (10 or 200/300) and compares full versus partial training, analyzing accuracy versus parameter fraction. Results reveal high redundancy, with performance degradation being minimal even when most weights are fixed.
Key Results
- On CIFAR-10, training only 10% of DenseNet parameters yields 76.33% top-1 accuracy, close to 78.5% with full training, indicating extreme redundancy.
- On CIFAR-100, similar parameter sparsity (e.g., 10%) maintains performance within 2-3% of full models, demonstrating robustness across datasets.
- On Tiny-ImageNet, training 70% of parameters results in only 2.3% accuracy loss, confirming scalability to larger datasets.
Significance
This work fundamentally challenges the conventional wisdom that neural networks require extensive parameter utilization for high performance. Revealing that most parameters are redundant opens new avenues for model compression, fast adaptation, and ensemble methods. It suggests that deep networks are overparameterized, and that training efficiency can be vastly improved without sacrificing accuracy, impacting both theoretical understanding and practical deployment of deep models.
Technical Contribution
The paper introduces a systematic framework to analyze parameter redundancy by selectively fixing network weights at random and training only subsets. It models the relationship between parameter fraction and accuracy as a logarithmic curve, supported by extensive experiments across architectures and datasets. This approach provides a new perspective on network overparameterization and offers practical strategies for efficient training and model compression, including partial layer or filter training and BN parameter tuning.
Novelty
This is the first comprehensive study to demonstrate that deep CNNs can perform nearly optimally when most parameters are fixed at random, extending beyond previous work on fixed random features or single-layer randomization. The combination of layer-wise slicing, filter subset training, and BN tuning across multiple architectures is novel, revealing deep network redundancy at an unprecedented scale.
Limitations
- The experiments focus primarily on image classification; applicability to other tasks like detection or segmentation remains unverified.
- Certain architectures (e.g., AlexNet) show less robustness to parameter sparsity, indicating architecture-dependent effectiveness.
- Large-scale training with fixed parameters still demands significant computational resources, and fixed parameters may limit fine-tuning flexibility.
Future Work
Future research will extend these findings to natural language processing and other domains, exploring automatic subset selection, integration with pruning and quantization, and theoretical analysis of parameter redundancy origins. Developing adaptive algorithms for optimal parameter subset identification and applying these insights to real-world deployment scenarios are promising directions.
AI Executive Summary
Deep neural networks have revolutionized many fields, yet their enormous parameter counts have raised questions about efficiency and redundancy. Traditionally, training involves updating all weights via backpropagation, which is computationally intensive and data-hungry. This study challenges that paradigm by demonstrating that fixing most parameters at random and only training a small subset can still yield performance comparable to fully trained models.
Using a systematic experimental framework, the authors tested various architectures—DenseNet, Wide-ResNet, VGG, AlexNet—on datasets like CIFAR-10, CIFAR-100, and Tiny-ImageNet. Surprisingly, training as little as 10% of the parameters in DenseNet achieved 76.33% accuracy on CIFAR-10, close to the 78.5% baseline. Similar results were observed across datasets, indicating a high degree of parameter redundancy. The relationship between the fraction of trained parameters and accuracy follows a logarithmic trend, suggesting diminishing returns beyond a certain point.
This discovery has profound implications. It indicates that deep networks are overparameterized, with many weights being redundant or interchangeable. Such insight can lead to more efficient training algorithms, model compression techniques, and robust ensemble methods that leverage fixed random backbones. The authors also explored selective layer and filter training, as well as tuning batch normalization parameters, further reducing training costs.
While promising, the approach has limitations, including its primary focus on image classification and the potential difficulty in applying fixed-parameter strategies to tasks requiring fine-grained adaptation. Nonetheless, this work opens new avenues for understanding deep learning's fundamental properties and optimizing model deployment in resource-constrained environments.
Deep Dive
Abstract
Training deep neural networks results in strong learned representations that show good generalization capabilities. In most cases, training involves iterative modification of all weights inside the network via back-propagation. In Extreme Learning Machines, it has been suggested to set the first layer of a network to fixed random values instead of learning it. In this paper, we propose to take this approach a step further and fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned. As our experiments show, fixing even the majority of the parameters of the network often results in performance which is on par with the performance of learning all of them. The implications of this intriguing property of deep neural networks are discussed and we suggest ways to harness it to create more robust representations.