Recursive Block-Diagonal Coupling for Resource-Efficient Training of Vision Models
Proposes Recursive Block-Diagonal Coupling (RBDC) for training wide vision models efficiently, reducing FLOPs by 30% while maintaining accuracy.
Key Findings
Methodology
RBDC employs a recursive approach where a target model of width W is decomposed into two narrower models of width W/2. Each narrow model is trained independently with different random seeds. The parameters are then combined into a wide model via a block-diagonal initialization with zero off-diagonal blocks, preserving the learned features. The process recursively continues until the narrowest models are trained from scratch. The training schedule uses a ratio r=2, allocating epochs proportionally to optimize resource use. Experiments on ImageNet with DeiT and ResNet architectures validate that this method reduces training FLOPs by up to 30% while maintaining or improving accuracy.
Key Results
- In ImageNet classification, DeiT-Base trained with one recursive step achieved approximately 83.5% accuracy with 30% fewer FLOPs compared to standard training. Two recursive steps further improved accuracy to 84.11%, with similar resource savings. ResNet-50D models also demonstrated significant efficiency gains, with better downstream task performance in object detection and segmentation tasks, confirming the method's robustness across architectures.
Significance
This work addresses the critical challenge of high computational costs in training large-scale vision models. By introducing a recursive training paradigm that leverages independently trained narrow models and a parameter-free initialization, it offers a scalable, resource-efficient alternative to traditional methods. The approach not only reduces training costs but also enhances the diversity of learned features, leading to better generalization and downstream performance. It paves the way for more sustainable AI development, especially in resource-constrained environments, and broadens the accessibility of large models in industry and academia.
Technical Contribution
The core innovation lies in the recursive width halving strategy combined with a block-diagonal zero-initialization scheme, enabling the construction of wide models from narrower, independently trained counterparts. Unlike prior model growth or pruning techniques, RBDC does not require pre-trained models or templates, making it universally applicable from scratch. The method introduces a training ratio parameter to balance resource allocation, supported by extensive ablation studies. Theoretically, it guarantees that the initialized wide model behaves as an ensemble of narrow models, facilitating efficient training without performance degradation.
Novelty
This is the first work to systematically integrate recursive width reduction with parameter-free block-diagonal initialization for training high-capacity vision models from scratch. Unlike existing model growth or ensembling methods, RBDC does not depend on pre-trained weights or templates, emphasizing a zero-initialization, recursive, from-scratch paradigm. Its recursive nature and resource-aware design distinguish it from prior approaches, offering a new theoretical and practical framework for resource-efficient deep learning.
Limitations
- The effectiveness of RBDC depends on the proper tuning of the training ratio r, which may vary across architectures and datasets. Its generalization to non-vision tasks remains to be validated.
- The block-diagonal initialization may have limited benefits for certain complex layers, such as attention modules with intricate interactions, requiring further adaptation.
- The recursive training process introduces additional complexity in scheduling and implementation, potentially limiting scalability in extremely large models or specialized hardware environments.
Future Work
Future research will explore adaptive strategies for training ratio adjustment, extend RBDC to multi-modal and multi-task settings, and integrate hardware-aware optimizations. Combining RBDC with low-precision training and distributed systems could further reduce costs, enabling scalable, sustainable training of ever-larger models. Additionally, theoretical analysis of feature diversity and ensemble effects within RBDC could deepen understanding and guide further improvements.
AI Executive Summary
The rapid growth of deep learning models, especially in computer vision, has led to unprecedented performance gains but also soaring computational costs. Training large models like Vision Transformers (DeiT) and ResNets from scratch often requires extensive resources, limiting accessibility and sustainability. Traditional methods such as pretraining, pruning, or incremental growth, while effective, still incur high costs and depend on pre-existing models or templates. Addressing this challenge, the paper introduces Recursive Block-Diagonal Coupling (RBDC), a novel training paradigm that constructs wide models recursively from narrower, independently trained sub-models.
RBDC leverages a recursive width halving strategy, where a target model of width W is decomposed into two narrower models of width W/2. Each narrow model is trained independently with distinct random seeds, capturing diverse features. These are then combined into a wide model via a parameter-free block-diagonal initialization, where off-diagonal blocks are zeroed out. This initialization ensures the wide model initially behaves as an ensemble of the narrow models, facilitating efficient training. The process repeats recursively, with a training schedule optimized by a ratio r=2, balancing resource allocation between narrow and wide phases.
Extensive experiments on ImageNet with DeiT and ResNet architectures demonstrate that RBDC reduces training FLOPs by approximately 30% while maintaining or surpassing baseline accuracy. For instance, DeiT-Base trained with one recursive step achieves about 83.5% accuracy, matching standard training costs but with significantly less computation. Two recursive steps further improve accuracy to 84.11%, confirming the method’s scalability. The approach also enhances downstream task performance, such as object detection and segmentation, showing its practical value.
This work offers a scalable, resource-efficient alternative to traditional training methods, with broad implications for AI sustainability. By enabling large models to be trained from scratch with reduced costs, RBDC opens new avenues for research and industrial deployment. Future directions include adaptive training schedules, multi-modal extensions, and hardware-aware optimizations, aiming to make high-capacity vision models more accessible and environmentally friendly.
Deep Dive
Abstract
Training high-capacity vision models from scratch requires substantial computational resources. To improve training efficiency of a wide target model, existing growth methods often assume the availability of narrower models, obscuring the true computational cost of the entire pipeline. We propose an efficient training protocol, RBDC, that builds wide models by coupling in a parameter-free block-diagonal way narrower, independently trained models in a recursive way. This allows a flexible allocation of the training budget available across all the models involved. Evaluated with vision transformers (DeiT) and convolutional networks (ResNet) on ImageNet, our RBDC training protocol shows a much better efficiency than models trained from scratch with the standard protocol, yielding 30% FLOPs reduction at similar test accuracies. It also achieves higher performances at same training FLOPs than training protocols from the model growth literature. Finally, we show that our models can serve as better backbones than their original counterparts for downstream object detection and instance segmentation tasks.