DynaBERT: Dynamic BERT with Adaptive Width and Depth

The evolution of NLP models from RNNs to Transformers has led to breakthroughs with models like BERT and RoBERTa, which achieve state-of-the-art results across numerous tasks. However, their massive parameter counts—often exceeding hundreds of millions—pose significant deployment challenges, especially on edge devices with limited memory, computation, and energy. Prior work on model compression, including knowledge distillation (DistilBERT, TinyBERT), pruning, quantization, and adaptive depth methods (LayerDrop), has mitigated some issues but largely produces fixed-size models. Recent efforts explore dynamic architectures, such as layer skipping or width pruning, but these are limited to single dimensions. The need for a flexible, multi-dimensional approach that can adapt to varying hardware constraints remains unmet, motivating the development of models like DynaBERT that can adjust both width and depth at inference time.

Despite advances, existing compression techniques lack the flexibility to adapt models dynamically during inference, which is crucial for real-world deployment on heterogeneous edge devices. Fixed-size models cannot efficiently utilize available resources, leading to either underutilization or performance degradation. The core challenge is designing a model that can flexibly trade off between accuracy and efficiency by adjusting its architecture in real-time, without retraining or fine-tuning for each configuration. Achieving this requires addressing issues like importance preservation of critical components, maintaining performance across configurations, and simplifying training complexity. Overcoming these bottlenecks is essential for making large NLP models practically deployable in resource-constrained environments.

The paper introduces DynaBERT, a framework that enables simultaneous width and depth adaptivity in Transformer models. Key innovations include: 1) importance-based network rewiring, which ranks attention heads and neurons to ensure critical components are shared across sub-networks; 2) a two-stage training process—first training a width-adaptive model via knowledge distillation, then extending to a full adaptive model; 3) multi-task distillation of logits, hidden states, and embeddings, combined with TinyBERT data augmentation; 4) a flexible architecture that allows multiple sub-network configurations, covering a broad spectrum of efficiency-performance trade-offs. These innovations collectively facilitate a scalable, versatile NLP model suitable for diverse deployment scenarios.

• �� Stage 1: Train DynaBERTW, a width-adaptive model, by ranking attention heads and FFN neurons based on importance scores derived from first-order Taylor expansion.
• �� Rewire network connections to prioritize important heads and neurons, ensuring shared critical components across sub-networks.
• �� Use knowledge distillation to transfer knowledge from the full teacher model to sub-networks with varying widths, aligning logits, embeddings, and hidden states.
• �� Incorporate TinyBERT data augmentation to enhance robustness.
• �� Stage 2: Use DynaBERTW as a teacher to train DynaBERT with both width and depth adaptivity, employing a similar distillation loss and layer dropping strategies for shallower sub-networks.
• �� Optimize the entire process with multi-task loss combining logits, hidden states, and embeddings, balancing performance across configurations.
• �� Finalize by optional fine-tuning on original data to boost accuracy.
• �� Validate across GLUE and SQuAD benchmarks, comparing with baselines and ablation variants.

Experiments utilize GLUE and SQuAD datasets, with BERT-base and RoBERTa-base as baselines. The training employs Nvidia V100 GPUs, exploring width ratios (1.0, 0.75, 0.5, 0.25) and depth ratios (1.0, 0.75, 0.5). Evaluation metrics include accuracy, F1, parameter count, FLOPs, and inference latency on GPU and ARM CPU. The models are compared against DistilBERT, TinyBERT, and LayerDrop, with ablation studies on network rewiring, distillation, and data augmentation. Performance is assessed under various efficiency constraints, demonstrating the trade-offs between size and accuracy. Extensive hyperparameter tuning ensures robustness and generalization across tasks.

The largest DynaBERT achieves accuracy comparable to BERT-base on GLUE, with over 30% reduction in parameters and FLOPs, and 40% faster inference on GPU and ARM CPU. Smaller configurations (e.g., 0.5x width/depth) still maintain 81.5% SQuAD EM, outperforming many compressed models. Ablation results confirm that importance-based rewiring and multi-task distillation significantly improve performance. Across tasks, DynaBERT consistently outperforms fixed-size compressed models like TinyBERT and LayerDrop, especially under strict efficiency constraints, validating the effectiveness of the adaptive approach.

DynaBERT is suitable for deployment in mobile NLP applications, real-time translation, and edge AI systems, where resource constraints vary dynamically. Its ability to adjust size on-the-fly allows optimal utilization of available hardware, reducing latency and energy consumption while maintaining high accuracy. It enables flexible, scalable NLP services across diverse devices without retraining, simplifying deployment pipelines and broadening accessibility.

Training complexity increases due to multi-stage procedures and importance ranking, raising development costs. Extreme compression may still cause performance drops, requiring further optimization. Hardware dependency during training limits accessibility for some users. Future work should focus on simplifying training, improving robustness at low resource levels, and extending to larger models and multi-modal tasks.