AdaDINO: Context-Adaptive DINO-Distilled Vision Foundation Models for Efficient Open-Vocabulary Edge Inference
AdaDINO uses task-adaptive NAS to improve edge vision models, boosting accuracy by 7.9% and reducing FLOPs by 74.9%.
Key Findings
Methodology
This work introduces a framework combining DINOv2-based vision models with neural architecture search (NAS) to create a family of subnets supporting task-dependent adaptive inference. Cloud-based multimodal large language models (LLMs) perform scene understanding, filtering candidate classes, and guiding subnet selection via a learned selector. The training involves three stages: NAS-based distillation of a supernet, vision-text alignment using contrastive loss, and training the subnet selector with scene-level class statistics. During inference, the system dynamically switches among subnets based on scene complexity, significantly reducing computation while maintaining accuracy.
Key Results
- On ImageNet-1K, AdaDINO reduces FLOPs by 74.9% while improving accuracy by 7.9%. On ADE20K open-vocabulary segmentation, it achieves a 5.2% increase in mIoU. The adaptive subnet selection reduces average compute by 37% at comparable accuracy, demonstrating superior efficiency-accuracy trade-offs.
- Across zero-shot classification and open-vocabulary segmentation, AdaDINO outperforms models of similar size, showing better accuracy-efficiency balance. Ablation studies reveal that joint contributions of distillation, scene understanding, and subnet selection are crucial for performance gains.
- Real-world deployment on edge hardware confirms the system’s robustness and energy efficiency, validating its practical utility in resource-constrained environments.
Significance
This research addresses the critical challenge of deploying large vision models on edge devices with limited resources. By enabling dynamic adaptation based on scene context, AdaDINO significantly reduces computational costs without sacrificing accuracy. It bridges the gap between high-performance AI and resource-constrained hardware, facilitating real-time, energy-efficient perception in applications like AR glasses, autonomous vehicles, and surveillance. The integration of cloud scene understanding with edge inference exemplifies effective end-to-end system design, setting a new paradigm for scalable edge AI solutions.
Technical Contribution
The paper introduces a novel combination of NAS-driven supernet training, vision-text alignment, and scene-aware subnet selection. It develops a multi-capacity supernet compatible with open-vocabulary tasks, trained via sandwich sampling. The learned selector predicts the optimal subnet based on scene class statistics, enabling real-time adaptive inference. This approach extends the capabilities of existing static models by allowing flexible capacity scaling, supported by theoretical guarantees on accuracy retention. It also demonstrates a practical deployment pipeline integrating cloud-based semantic understanding with edge inference, advancing the state-of-the-art in resource-efficient vision models.
Novelty
This is the first work to integrate NAS with open-vocabulary vision models for task-dependent adaptive inference on edge devices. Unlike prior static compression or confidence-based methods, AdaDINO dynamically switches among multiple subnets conditioned on scene semantics, achieving a superior accuracy-efficiency trade-off. Its combination of scene understanding, NAS, and learned subnet selection represents a significant innovation in scalable edge AI, enabling models to adapt to varying scene complexities in real time.
Limitations
- The system heavily relies on cloud-based scene understanding, which may be vulnerable to network latency or failure, impacting real-time performance.
- Training the NAS supernet and subnet selector involves high computational costs and complex pipelines, limiting ease of deployment.
- The approach assumes scene stability over several frames, which may not hold in highly dynamic environments, potentially reducing adaptation effectiveness.
Future Work
Future directions include developing more efficient scene understanding modules to reduce cloud dependency, enhancing the generalization of subnet selection across diverse scenarios, and exploring federated learning approaches to enable decentralized training. Additionally, integrating more advanced multi-modal understanding and extending the framework to other vision tasks like detection and tracking could further broaden its applicability.
AI Executive Summary
In recent years, the deployment of large-scale vision foundation models (VFMs) has revolutionized AI capabilities across tasks such as classification and segmentation. However, their immense computational demands pose a significant barrier for edge devices like AR glasses, autonomous drones, and mobile robots. Traditional approaches to model compression—such as pruning, quantization, or static distillation—offer some relief but often lead to substantial accuracy loss, especially when models are scaled down for resource constraints.
This paper introduces AdaDINO, a novel framework that dynamically adapts the capacity of vision models based on scene complexity and task requirements. By integrating neural architecture search (NAS) into a DINOv2-based backbone, AdaDINO constructs a supernet supporting multiple subnets of varying sizes. During inference, a cloud-based multimodal large language model (LLM) performs scene understanding, filtering candidate classes and providing semantic context. A learned subnet selector then chooses the smallest subnet capable of maintaining a target accuracy level, significantly reducing computational load.
The core innovation lies in the task-aware, scene-conditioned adaptation mechanism. This approach allows the system to allocate resources efficiently, spending more on complex scenes and less on simpler ones. Experimental results on datasets like IN1K and ADE20K demonstrate that AdaDINO achieves up to 7.9% higher accuracy and reduces FLOPs by 74.9%, outperforming comparable models. The system’s robustness and efficiency are validated through deployment on ARM Ethos-U55 hardware, confirming its practical value.
Overall, AdaDINO represents a significant step forward in scalable edge AI, combining advanced neural architecture search, scene understanding, and dynamic model selection. Its ability to balance accuracy and efficiency in real-time opens new avenues for intelligent, energy-efficient perception in resource-limited environments. Future work will focus on further reducing cloud dependency and extending adaptability to more complex multi-modal tasks, promising a versatile framework for next-generation edge AI systems.
Deep Dive
Glossary
Neural Architecture Search (NAS)
一种自动设计神经网络结构的方法，通过搜索空间找到最优架构。In this paper, NAS is used to generate a supernet supporting multiple subnets for adaptive inference.
用于训练支持多容量子网的超网，提升模型适应性。
DINOv2
一种基于自监督学习的视觉基础模型，能学习具有良好迁移能力的特征表示。本文利用其蒸馏版本作为基础架构。
作为模型蒸馏的目标模型，提供强大特征提取能力。
Open-Vocabulary
支持无限类别或词汇的任务，不依赖固定标签集。本文通过云端场景理解和文本对齐实现。
实现零样本分类和开放词汇分割。
Contrastive Learning
一种通过最大化正样本对相似度、最小化负样本对相似度的学习方法。用于视觉文本对齐。
训练视觉特征与文本特征的对齐模型。
Sandwich Sampling
一种训练策略，交替采样模型的最大和最小子网以平衡性能。本文用于NAS超网训练。
确保多子网模型的性能一致性。
Open Questions Unanswered questions from this research
- 1 如何进一步降低云端依赖，提升系统在网络不稳定环境下的鲁棒性。当前方法在极端动态场景中的适应能力仍需验证。未来应探索端端自主场景理解与模型自适应机制。
Applications
Immediate Applications
智能监控
在监控摄像头中部署AdaDINO，实现实时场景识别和异常检测，节省能耗同时保证监控效果。
增强现实
在AR眼镜中应用，支持实时场景理解和对象识别，提升用户交互体验，延长电池续航。
Long-term Vision
智慧城市
实现城市环境的智能感知与管理，支持交通监控、公共安全等多场景应用，推动城市智能化升级。
Abstract
Always-on contextual AI runs language-aligned vision foundation models (VFMs) on edge devices, where the on-device model is the dominant continuous compute cost under strict latency and power limits. Due to an observed low-frequency shift in scene context and its relevant vocabulary, we present AdaDINO, an adaptive framework that makes on-device VFM inference efficient by matching execution to the current scene and task. We build on a known phenomenon, that the accuracy drop of shrinking model sizes depends on the task, and turn it into task-level adaptive execution. AdaDINO integrates neural architecture search (NAS) into a language-aligned VFM backbone distilled from DINOv2, training a single family of subnets for efficient execution during runtime. A multimodal large language model (LLM) on the cloud, invoked at low frequency, refines the candidate class set from scene context, while a learned selector activates the least-cost subnet predicted to retain a target fraction of accuracy. With the backbone and semantic pipeline held fixed, learned selection alone reduces average compute by $37\%$ over the best fixed subnet at equal segmentation accuracy. Across zero-shot classification and open-vocabulary segmentation, AdaDINO establishes a strong accuracy-efficiency frontier, improving over evaluated models of comparable sizes by up to $7.9\%$ in acc@1 on IN1K and $5.2\%$ mIoU on ADE20K, and reducing average FLOPs by up to $74.9\%$ at similar accuracy.