Open-vocabulary Object Detection via Vision and Language Knowledge Distillation
ViLD employs vision-language knowledge distillation to enable open-vocabulary object detection, achieving 16.1 mask AP_r on LVIS.
Key Findings
Methodology
This paper introduces ViLD, which integrates pre-trained vision-language models like CLIP into two-stage detectors via knowledge distillation. The approach involves encoding category texts and image regions using a teacher model, then training the detector's region embeddings to align with these embeddings. ViLD employs two strategies: ViLD-text uses text embeddings for classification, while ViLD-image distills visual features from precomputed image embeddings. The training optimizes cross-entropy for classification and L1 loss for embedding alignment, enabling the detector to recognize unseen categories. Experiments on LVIS, holding out rare classes, show that ViLD achieves 16.1 mask AP_r, surpassing supervised baselines by 3.8. Using the stronger ALIGN teacher boosts performance to 26.3. The model generalizes well to other datasets like PASCAL VOC, COCO, and Objects365 without fine-tuning, demonstrating robust transferability.
Key Results
- On LVIS, ViLD with ResNet-50 backbone achieves 16.1 mask AP_r, outperforming supervised methods by 3.8, and reaches 26.3 AP_r with ALIGN, nearing state-of-the-art.
- Transfer results show 72.2 AP50 on PASCAL VOC, 36.6 AP on COCO, and 11.8 AP on Objects365, all without dataset-specific fine-tuning.
- On COCO, ViLD exceeds previous SOTA by 4.8 on novel AP and 11.4 overall AP, confirming its strong generalization to diverse datasets.
Significance
This work addresses the long-standing challenge of scaling object detection vocabularies without extensive data collection. By leveraging abundant web-scale image-text pairs and pre-trained models, ViLD enables zero-shot detection for thousands of categories, reducing reliance on costly annotations. Its ability to transfer across datasets and recognize novel objects has significant implications for real-world applications such as autonomous driving, robotics, and large-scale image indexing, where categories are dynamic and annotations are scarce. The approach marks a step toward truly open-world visual understanding, bridging the gap between image-level recognition and object detection.
Technical Contribution
The key technical innovation lies in integrating vision-language pre-trained models into the detection pipeline through knowledge distillation. The paper introduces separate distillation pathways—ViLD-text for semantic classification and ViLD-image for visual feature alignment—using cosine similarity and L1 loss, respectively. This flexible framework allows the detector to learn from rich, multi-modal embeddings without requiring explicit annotations for each category. The architecture supports different teacher models, such as CLIP and ALIGN, and employs ensembling techniques to further boost performance. These contributions significantly improve open-vocabulary detection accuracy, especially on rare classes, and demonstrate the effectiveness of multi-modal knowledge transfer in object detection.
Novelty
This is the first work to effectively incorporate large-scale vision-language pre-trained models into object detection via knowledge distillation, enabling open-vocabulary detection beyond traditional supervised datasets. Unlike prior zero-shot detection methods limited to small category sets, ViLD scales to over 1,000 categories, leveraging multi-modal embeddings for both classification and localization. Its dual distillation strategy and flexible architecture represent a novel paradigm shift, moving from closed-set detection to a more general, scalable framework that can recognize arbitrary text descriptions.
Limitations
- Inference speed remains slow due to per-region embedding computation, limiting real-time deployment.
- Performance on extremely rare or highly imbalanced categories still lags, indicating room for better few-shot or semi-supervised learning strategies.
- Dependence on pre-trained models like CLIP or ALIGN means that if these models degrade or are unavailable, the detection performance may suffer.
Future Work
Future directions include optimizing inference efficiency, possibly through embedding caching or model pruning; enhancing few-shot learning capabilities for ultra-rare categories; and integrating more advanced multi-modal fusion techniques. Extending the framework to video detection and real-time applications, as well as exploring unsupervised or self-supervised pre-training for even broader vocabulary coverage, are promising avenues. Additionally, further research could focus on reducing reliance on specific pre-trained models, making the approach more adaptable and accessible across different domains.
AI Executive Summary
Object detection has traditionally relied on extensive labeled datasets, limiting its ability to recognize the ever-expanding variety of objects encountered in real-world scenarios. As the number of categories grows, collecting sufficient annotations becomes prohibitively expensive. Recent advances in vision-language models like CLIP and ALIGN have demonstrated remarkable zero-shot recognition capabilities at the image level, but their integration into object detection remains challenging.
This paper introduces ViLD, a novel framework that bridges this gap through knowledge distillation. By leveraging pre-trained vision-language models, ViLD encodes category texts and image regions into a shared semantic space. The core idea is to distill this multi-modal knowledge into a two-stage object detector, enabling it to recognize arbitrary text descriptions without explicit training on each category.
The methodology involves generating text embeddings for categories using models like CLIP, precomputing image embeddings for object proposals, and training the detector's region embeddings to align with these. Two strategies are employed: ViLD-text, which uses text embeddings for classification, and ViLD-image, which distills visual features from the precomputed image embeddings. Combining these, the detector learns to localize and classify objects across a vast vocabulary.
Experimental results on LVIS show that ViLD achieves 16.1 mask AP_r on held-out rare categories, surpassing supervised baselines by 3.8. When using the stronger ALIGN teacher, performance improves to 26.3. The model also transfers effectively to datasets like PASCAL VOC, COCO, and Objects365, with high AP scores without fine-tuning. These results demonstrate the approach’s robustness and scalability.
Overall, ViLD marks a significant step toward open-world object detection, reducing dependency on costly annotations and enabling recognition of a limitless set of categories. Its innovative use of multi-modal knowledge distillation opens new avenues for scalable, flexible, and intelligent visual systems. Future work will focus on improving inference speed, handling extremely rare categories, and extending to video and real-time applications, promising a transformative impact on computer vision.
Deep Dive
Abstract
We aim at advancing open-vocabulary object detection, which detects objects described by arbitrary text inputs. The fundamental challenge is the availability of training data. It is costly to further scale up the number of classes contained in existing object detection datasets. To overcome this challenge, we propose ViLD, a training method via Vision and Language knowledge Distillation. Our method distills the knowledge from a pretrained open-vocabulary image classification model (teacher) into a two-stage detector (student). Specifically, we use the teacher model to encode category texts and image regions of object proposals. Then we train a student detector, whose region embeddings of detected boxes are aligned with the text and image embeddings inferred by the teacher. We benchmark on LVIS by holding out all rare categories as novel categories that are not seen during training. ViLD obtains 16.1 mask AP$_r$ with a ResNet-50 backbone, even outperforming the supervised counterpart by 3.8. When trained with a stronger teacher model ALIGN, ViLD achieves 26.3 AP$_r$. The model can directly transfer to other datasets without finetuning, achieving 72.2 AP$_{50}$ on PASCAL VOC, 36.6 AP on COCO and 11.8 AP on Objects365. On COCO, ViLD outperforms the previous state-of-the-art by 4.8 on novel AP and 11.4 on overall AP. Code and demo are open-sourced at https://github.com/tensorflow/tpu/tree/master/models/official/detection/projects/vild.