Deep Fried Convnets

Deep CNNs, exemplified by AlexNet, VGG, and ResNet, have achieved remarkable success in visual recognition tasks. However, their massive parameter counts, especially in fully connected layers, pose challenges for deployment on devices with limited memory and computational power. Early efforts like parameter sharing, pruning, and low-rank factorization partially mitigated these issues but often at the cost of reduced accuracy or increased complexity. Kernel methods and random feature approximations, such as Random Fourier Features, have been explored to enhance efficiency. Fastfood transform emerged as an efficient structured random projection, reducing costs from O(nd) to O(n log d). Integrating these ideas into deep learning, especially for fully connected layers, remains an ongoing challenge, motivating the development of learnable, end-to-end trainable parameter-efficient modules.

The core challenge is to drastically reduce the number of parameters in fully connected layers of CNNs without sacrificing predictive accuracy. Existing methods like low-rank approximation and pruning are either post hoc or limited in compression ratio. Moreover, dense softmax layers in large-scale datasets like ImageNet dominate parameter budgets, hindering model deployment on resource-constrained devices. Achieving a balance between model compactness, training efficiency, and performance robustness remains unresolved, especially for end-to-end trainable solutions that can adapt during training.

This work introduces an adaptive Fastfood transform that replaces dense fully connected layers with a structured, learnable projection. Key innovations include: 1) Parameterizing the diagonal matrices S, G, B within the transform, enabling end-to-end learning via backpropagation; 2) Utilizing Walsh-Hadamard transforms to achieve computational complexity of O(n log d); 3) Extending the transform to large output layers by stacking multiple modules; 4) Demonstrating that the transform preserves kernel properties, allowing for effective feature approximation. These innovations collectively enable significant parameter reduction while maintaining accuracy, representing a new paradigm in neural network compression.

• �� Input features from convolutional layers serve as input to the Fastfood layer. • The core module: input passes through a Walsh-Hadamard transform (H), then multiplies element-wise with learned diagonal matrices S, G, B, and a permutation matrix Π. • The output is obtained by applying the inverse Walsh-Hadamard transform, followed by a nonlinear activation. • Parameters S, G, B are initialized randomly and learned via backpropagation, with gradients computed efficiently using the Fast Walsh-Hadamard transform. • The transform is extended to larger output dimensions by stacking multiple modules, maintaining O(n log d) complexity. • The entire network, including convolutional and Fastfood layers, is trained jointly, optimizing for classification accuracy. • Hyperparameters like feature count are tuned to balance compression and performance. • Additional techniques like dropout are employed to prevent overfitting during training.

Experiments on MNIST and ImageNet validate the approach. On MNIST, replacing the fully connected layer with 1024 features reduces parameters from 430,500 to 38,821, with error rate stable at 0.83%. Increasing features to 2048 yields similar accuracy with fewer parameters. On ImageNet, the baseline model with 58.7M parameters is compressed to 16.4M using 16,384 features, with only a 0.3% increase in top-1 error. Further increasing features to 32,768 reduces parameters to 32.8M, with a slight accuracy gain. Fixed feature extraction experiments show that adaptive Fastfood improves performance over non-adaptive variants, especially in large-scale datasets. The results demonstrate that the method scales well and maintains competitive accuracy while drastically reducing model size.

The method achieves over 90% parameter reduction in CNNs with negligible accuracy loss. On MNIST, error rates remain below 0.83%. On ImageNet, parameters are reduced by over 70%, with top-1 error increasing marginally by 0.3%. Adaptive parameter learning enhances performance, outperforming non-adaptive and traditional low-rank methods. The results confirm the effectiveness of structured random projections for deep network compression, with potential for further improvements via multi-layer stacking and hybrid techniques.

This technique enables deployment of deep CNNs on mobile devices, embedded systems, and edge hardware, facilitating real-time image recognition, video analysis, and autonomous systems. It reduces memory footprint and computational load, making AI more accessible and energy-efficient. In industry, it can be integrated into existing frameworks for model compression, transfer learning, and rapid deployment, especially where hardware constraints are critical. Long-term, this approach paves the way for ubiquitous AI in everyday devices, supporting scalable, low-cost intelligent systems.

While highly effective, the method still relies on dense softmax layers for large class counts, limiting overall compression in some scenarios. The learnability of transform parameters may degrade under extreme compression ratios, affecting convergence stability. Hyperparameter sensitivity requires careful tuning, and the method's performance on tasks beyond image classification remains to be validated. Future work should address multi-layer parameterization, robustness, and integration with other compression techniques to overcome these limitations.