Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure

The rapid development of deep CNNs has led to models with hundreds of millions of parameters, demanding immense computational resources. Early efforts focused on pruning connections or neurons, but these often resulted in sparse models that lacked speedup without specialized hardware. Filter pruning emerged as a practical solution, directly reducing FLOPs by removing entire channels. Techniques like importance ranking based on Taylor expansion, magnitude, or information-theoretic measures have been proposed, but they typically require iterative fine-tuning to recover accuracy. Complex architectures such as ResNet and DenseNet introduce layer interdependencies, making layer-wise pruning difficult. Recent advances include regularization-based zeroing methods and structured pruning, yet they still face limitations in handling constraints across layers. The challenge remains to develop a universal, efficient, and constraint-aware pruning method that preserves performance while significantly reducing model size.

The core problem is how to prune deep, complex CNNs with structural dependencies, such as residual and dense connections, without performance degradation. Existing importance-based methods often fail to handle inter-layer pruning constraints, requiring layer-by-layer approaches that are inefficient and prone to error accumulation. Zeroing methods reduce parameters but necessitate costly fine-tuning, limiting their practicality. The key difficulty lies in achieving layer-wise and inter-layer consistent pruning, especially in networks with intricate connectivity. Moreover, ensuring that the pruned model retains the original representational capacity while being significantly smaller remains an open challenge. Addressing these issues requires a method that can enforce redundancy patterns during training, enabling direct, constraint-aware pruning.

The main innovation is the introduction of C-SGD, which enforces a centripetal constraint during training, causing filters within the same cluster to converge to a common point in parameter space. This is achieved by modifying the SGD update rule to include a centripetal force term, which pulls filters together. The algorithm employs a cluster-based approach, where filters are grouped either randomly or via k-means, and updates are performed in a matrix-efficient manner. This process ensures that, after training, filters in each cluster are identical, allowing for direct merging and pruning without fine-tuning. The method also extends to constrained pruning scenarios, such as in residual networks, by enforcing shared redundancy patterns across layers. Theoretically, it demonstrates how redundancy can be exploited to improve convergence and model compression simultaneously.

• �� Cluster formation: Divide filters into groups via k-means or even grouping, based on their flattened parameters. • Loss augmentation: Add a centripetal regularization term to the original loss, encouraging filters within each cluster to converge. • Gradient update: During each iteration, compute the average gradient within each cluster and update filters by combining the original gradient with a centripetal force proportional to the difference between cluster center and filters. • Efficient implementation: Reshape filter tensors into matrices, construct averaging and decay matrices, and perform updates via matrix multiplication to maintain computational efficiency. • Training process: Optimize the combined loss to gradually merge filters within clusters, maintaining performance. • Pruning: Post-training, select one representative filter per cluster, delete others, and merge corresponding input channels in subsequent layers, leveraging convolution's linearity for no performance loss.

Experiments conducted on CIFAR-10 and ImageNet validate C-SGD's effectiveness. On CIFAR-10, models like ResNet-56 and DenseNet-40 were trained with approximately 60% FLOPs reduction, with accuracy drops less than 0.3%. On ImageNet, ResNet-50 with C-SGD achieved similar FLOP reduction, with only 0.06% accuracy loss. The experiments compare different clustering strategies, showing robustness to random or k-means clustering. Additional ablation studies analyze the influence of hyperparameters like centripetal strength. The results confirm that C-SGD can produce highly compressed models with minimal performance degradation, outperforming traditional importance-based pruning and zeroing methods.

C-SGD achieves around 60% FLOPs reduction on ResNet-50 with only 0.06% accuracy loss on ImageNet, outperforming importance-based pruning. In DenseNet-40, the method not only compresses the model but also improves accuracy, validating the redundancy hypothesis. The experiments demonstrate that filters trained with C-SGD converge to identical parameters, enabling direct pruning. The robustness across different clustering strategies and network types highlights its practical value. Additionally, training redundant models with C-SGD yields better convergence and generalization than standard training, supporting the idea that redundancy benefits training stability.

This technique is suitable for deploying deep CNNs on resource-constrained devices such as smartphones, embedded systems, and IoT platforms. It enables automatic, constraint-aware model compression, reducing computational load and memory footprint without sacrificing accuracy. The approach can be integrated into existing training pipelines, facilitating efficient model deployment in real-world applications like autonomous vehicles, surveillance, and medical imaging. Future integration with hardware-aware optimization can further enhance inference speed and energy efficiency.

The method depends on hyperparameters like centripetal strength, which require tuning for different architectures and tasks. Its effectiveness diminishes if the network has minimal redundancy or highly non-linear filter relationships. Extreme compression ratios may lead to slight accuracy drops, especially in sensitive layers. Additionally, the approach currently focuses on convolutional filters and may need adaptation for other architectures such as transformers or graph neural networks. Further research is needed to automate parameter selection and extend applicability.