EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks
Proposes a compound scaling method balancing depth, width, and resolution, achieving 84.3% top-1 accuracy on ImageNet with 8.4x fewer parameters.
Key Findings
Methodology
This paper systematically investigates CNN scaling, discovering relationships among depth, width, and resolution. It introduces a compound coefficient φ, with hyperparameters α, β, γ, to uniformly scale these dimensions. Validated on MobileNets and ResNet, combined with neural architecture search, the approach yields EfficientNet models. Experiments show EfficientNet-B7 achieves 84.3% Top-1 accuracy on ImageNet, with parameters reduced by 8.4x and inference speed increased by 6.1x, outperforming prior models significantly.
Key Results
- EfficientNet-B7 reaches 84.3% Top-1 accuracy on ImageNet, surpassing GPipe by 0.0%, with 8.4x fewer parameters and 6.1x faster inference.
- On transfer datasets like CIFAR-100 and Flowers, EfficientNets outperform ResNet and MobileNets, with parameter reductions up to 20x and accuracy gains of 1-2%.
- Applying the scaling to existing architectures like MobileNets and ResNet demonstrates broad applicability, with consistent improvements in efficiency and accuracy.
Significance
This work addresses the longstanding challenge of model scaling by providing a principled, systematic approach that balances multiple dimensions. It advances the field by enabling high-performance models with minimal parameters, crucial for deploying deep learning in resource-constrained environments. The methodology bridges theoretical insights and practical engineering, fostering automated, scalable model design, and broadening AI's real-world impact.
Technical Contribution
The key contribution is the formulation of a compound scaling method using a single coefficient φ, with hyperparameters α, β, γ, to jointly scale depth, width, and resolution. This approach is grounded in empirical analysis and theoretical reasoning, enabling models to grow efficiently. The integration with neural architecture search to design the EfficientNet baseline further enhances the method's effectiveness, setting a new standard for scalable CNN design. The approach offers a clear, mathematically grounded framework for model scaling, outperforming traditional heuristic methods.
Novelty
This is the first work to systematically quantify and unify the scaling of depth, width, and resolution through a compound coefficient, providing a theoretical and empirical foundation. Unlike prior heuristic or manual tuning, this method offers a principled, automated approach. The integration of neural architecture search with this scaling strategy results in a family of highly efficient, high-accuracy models, marking a significant innovation in model design methodology.
Limitations
- The method relies on a predefined baseline architecture, and its effectiveness may vary across different network designs. Hyperparameter tuning, although simplified, still requires some manual effort.
- While efficient, the approach may face challenges in extremely resource-constrained environments where model complexity must be further reduced, possibly at the expense of accuracy.
- The current framework does not explicitly incorporate hardware-specific constraints, which could further optimize deployment in diverse real-world scenarios.
Future Work
Future research will focus on automating hyperparameter selection, integrating hardware-aware optimization, and extending the approach to other tasks like object detection and segmentation. Additionally, combining this scaling method with model compression techniques such as pruning and quantization could further enhance efficiency. Exploring adaptive scaling strategies based on real-time resource availability is also a promising direction.
AI Executive Summary
Deep convolutional neural networks have revolutionized image recognition, yet their increasing size and complexity pose practical challenges for deployment, especially on resource-limited devices. Traditional scaling strategies—such as simply adding more layers or increasing input resolution—lack a unified theoretical framework, often leading to suboptimal trade-offs between accuracy and efficiency. This paper introduces a novel compound scaling method that simultaneously adjusts depth, width, and resolution using a single coefficient φ, guided by empirically determined hyperparameters α, β, and γ. This approach ensures a balanced, systematic expansion of models, enabling significant improvements in performance and efficiency.
By applying this method to MobileNets and ResNet, and further refining with neural architecture search, the authors develop EfficientNet models. Notably, EfficientNet-B7 achieves 84.3% Top-1 accuracy on ImageNet, surpassing the previous state-of-the-art GPipe while using 8.4 times fewer parameters and running 6.1 times faster. These models also transfer well to other datasets like CIFAR-100 and Flowers, maintaining high accuracy with substantially fewer parameters.
The significance of this work lies in its principled approach to model scaling, providing a clear framework that can be universally applied across architectures. It addresses the core challenge of balancing model capacity and efficiency, crucial for deploying AI in real-world scenarios. The integration with neural architecture search further enhances the method's robustness and adaptability.
Looking ahead, future work will focus on automating hyperparameter tuning, incorporating hardware-specific constraints, and extending the approach to other vision tasks. Combining this scaling strategy with model compression techniques promises even greater efficiency, paving the way for scalable, high-performance AI systems suitable for diverse applications and devices.
Deep Dive
Abstract
Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performance. Based on this observation, we propose a new scaling method that uniformly scales all dimensions of depth/width/resolution using a simple yet highly effective compound coefficient. We demonstrate the effectiveness of this method on scaling up MobileNets and ResNet. To go even further, we use neural architecture search to design a new baseline network and scale it up to obtain a family of models, called EfficientNets, which achieve much better accuracy and efficiency than previous ConvNets. In particular, our EfficientNet-B7 achieves state-of-the-art 84.3% top-1 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet. Our EfficientNets also transfer well and achieve state-of-the-art accuracy on CIFAR-100 (91.7%), Flowers (98.8%), and 3 other transfer learning datasets, with an order of magnitude fewer parameters. Source code is at https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet.