CoSeP: Complementary Separability Pruning via Class-Separability Clustering

Deep neural networks excel in visual tasks but are often too large for practical deployment, especially on edge devices. Early methods like weight pruning and quantization reduced model size but required manual tuning and often compromised accuracy. Structured pruning, which removes entire filters or neurons, gained popularity for hardware efficiency. Notable techniques include L1-norm pruning, FPGM, and ThiNet, which focus on importance scores. However, these methods typically treat components independently, ignoring potential redundancy and complementarity. As models grow deeper and more complex, the need for automatic, principled pruning strategies becomes critical, prompting research into methods that can adaptively identify redundant components without extensive manual intervention.

Existing pruning techniques primarily rely on importance scores, such as weight magnitude or activation levels, to select components for removal. This approach neglects the functional redundancy among components—multiple neurons or filters may encode similar discriminative features, leading to inefficient compression. Additionally, manually setting pruning ratios per layer is labor-intensive and suboptimal, as different layers exhibit varying redundancy and sensitivity. These issues hinder the scalability and effectiveness of model compression, especially for large-scale architectures and datasets. The core challenge is to develop an automated, geometry-based method that can identify redundant components and determine optimal pruning levels intrinsically, without extensive hyperparameter tuning or external search strategies.

The key innovation of CoSeP is embedding each component into a class-separability space defined by Jeffries–Matusita distances across class pairs. This space encodes the functional discriminability of components, with proximity indicating redundancy and distance indicating complementarity. By applying k-medoids clustering, the method groups components with similar discriminative profiles, ensuring diverse feature coverage. The use of the Mean Silhouette index and Kneedle algorithm allows automatic detection of the optimal number of components to retain, removing the need for manual ratio specification. This geometric, data-driven approach enables a more accurate and automatic identification of redundant components, improving compression efficiency while maintaining or enhancing accuracy.

• �� Construct class-separability profiles: For each component, compute JM distances across selected class pairs, forming a high-dimensional feature vector.
• �� Map components into a geometric space: Use Euclidean distance to measure similarity between components based on their profiles.
• �� Cluster components: Apply k-medoids clustering for a range of cluster counts, evaluating each with MSS.
• �� Determine optimal cluster count: Fit a polynomial to MSS curve, detect the knee point with Kneedle, and select this as the number of components to retain.
• �� Select representatives: For each cluster, choose the component with the largest parameter norm.
• �� Layer-wise pruning: Retain selected components, prune others, and fine-tune the network.
• �� Repeat for all prunable layers sequentially, ensuring minimal manual intervention.

The experiments involve CIFAR-10, CIFAR-100, and ImageNet-1K datasets, testing architectures like ResNet, VGG, MobileNet, and DenseNet. Baselines include random pruning and importance-score methods. Metrics focus on Top-1 accuracy, FLOPs reduction, and inference time. Hyperparameters include M=100 for class selection and polynomial degree p=2 for MSS smoothing. Multiple runs ensure statistical robustness. Fine-tuning after each layer is performed with 2-3 epochs, maintaining performance. The evaluation demonstrates that CoSeP consistently outperforms or matches existing methods, with significant reductions in FLOPs and inference time while preserving accuracy.

On ResNet-50/ImageNet, CoSeP achieves +0.66% accuracy gain with 2.30× FLOPs reduction; VGG-16/CIFAR-10 sees +0.37% gain at 2.59× reduction. Across architectures, it maintains or improves accuracy while significantly reducing computational cost. The automatic ratio determination aligns well with the intrinsic redundancy structure, outperforming fixed-ratio or importance-based methods. Results show robustness across datasets and models, validating the effectiveness of class-separability space clustering in neural network pruning.

This method is suitable for deploying large models on resource-limited devices like smartphones and embedded systems. It requires only a small labeled calibration set, making it practical for real-world scenarios. The automatic determination of pruning ratios simplifies model compression workflows, reducing manual tuning. Future applications include multi-task learning, multi-modal models, and integration with other compression techniques like quantization, broadening the scope of efficient deep learning deployment.

The approach relies on labeled data for calibrating class-separability profiles, limiting its use in unlabeled or semi-supervised settings. Its effectiveness diminishes with extremely large class sets or severe class imbalance. The fine-tuning process adds computational overhead, and hyperparameters such as class subset size M need careful tuning. Further research is needed to extend the method to unsupervised or semi-supervised contexts and to optimize clustering efficiency for very large models.