WeDetect: Fast Open-Vocabulary Object Detection as Retrieval
WeDetect achieves fast open-vocabulary object detection via retrieval, achieving SOTA across 15 benchmarks with high inference efficiency.
Key Findings
Methodology
WeDetect employs a dual-tower architecture without cross-modal fusion layers, treating object detection as a retrieval problem. Leveraging CLIP pretraining, a ConvNeXt backbone, and a high-quality dataset, it achieves superior performance.
Key Results
- WeDetect-Tiny achieves 37.4 AP on LVIS minival, outperforming YOLO-World-L by 2.0 AP, with inference speed of 62.5 fps.
- WeDetect-Large achieves 55.0 AP on LVIS, surpassing T-Rex2 by 3.6 AP, while being 3× faster.
- WeDetect-Uni outperforms CLIP by 37.2 F1 in object retrieval tasks, showcasing fine-grained perception advantages.
Significance
This research addresses efficiency bottlenecks in open-vocabulary detection, unifying detection, proposal generation, object retrieval, and complex expression comprehension, advancing vision-language applications.
Technical Contribution
Introduces a fusion-free dual-tower architecture for faster inference; develops a high-quality data engine with 15M images and 330M bounding boxes; designs an LLM-based object classifier for complex expression comprehension.
Novelty
First to fully reformulate open-vocabulary detection as a retrieval problem, proposing a unified framework (WeDetect family) covering detection, proposal generation, and complex expression comprehension.
Limitations
- Small object detection performance remains suboptimal in certain scenarios.
- Relies heavily on large-scale pretrained models, demanding significant computational resources.
- Annotation quality from the data engine may be limited by model capabilities.
Future Work
Future research could explore more efficient architectures, improve small object detection, expand data engine capabilities, and study cross-domain generalization.
AI Executive Summary
Open-vocabulary object detection aims to detect arbitrary categories via text prompts, but existing methods suffer from inefficiency and limited scalability.
WeDetect introduces a unified retrieval-based framework using a dual-tower architecture without cross-modal fusion layers. By leveraging CLIP pretraining, a ConvNeXt backbone, and a curated dataset, it achieves superior detection performance and inference speed. The WeDetect family includes the base model, WeDetect-Uni for universal proposal generation, and WeDetect-Ref for complex expression comprehension.
Experiments demonstrate SOTA performance across 15 benchmarks, including LVIS and COCO, with significantly faster inference. This unified framework addresses key challenges in open-vocabulary detection, object retrieval, and complex expression understanding, advancing vision-language research.
Deep Analysis
Background
Open-vocabulary object detection has gained attention for its ability to detect arbitrary categories via text prompts. Traditional methods rely on deep cross-modal fusion layers, which, while accurate, suffer from inefficiency and scalability issues.
Core Problem
Existing methods face slow inference speeds when handling large vocabularies, e.g., Grounding-DINO requires 31 forward passes on LVIS, causing several seconds of latency per image. Fusion layers also prevent feature sharing across queries, further limiting efficiency.
Innovation
WeDetect proposes a fusion-free dual-tower architecture, reformulating detection as a retrieval problem. It leverages CLIP pretraining, a ConvNeXt backbone, and a high-quality dataset to achieve superior performance and efficiency.
Methodology
- �� Dual-tower architecture: Language encoder (XLM-RoBERTa) and visual encoder (ConvNeXt) work independently; classification via dot product between grid features and text embeddings.
- �� Data engine: Collects 15M images, generates 330M bounding boxes using SAM and Qwen2.5-VL for multi-granularity labels.
- �� WeDetect-Uni: Freezes the detector, trains a universal objectness prompt for high-recall proposal generation.
- �� WeDetect-Ref: LLM-based classifier processes complex expressions in a single forward pass.
Experiments
Experiments use LVIS, COCO, ODinW benchmarks to evaluate open-vocabulary detection. Baselines include YOLO-World-L, Grounding-DINO. Additional tests include object retrieval and complex expression comprehension.
Results
WeDetect achieves 55.0 AP on LVIS, 3× faster inference than Grounding-DINO; object retrieval task shows 37.2 F1 improvement over CLIP; complex expression comprehension achieves 93.2 average accuracy.
Applications
Applicable to real-time object detection, content moderation, and keyword-based image retrieval, especially for large-scale categories and complex expressions.
Limitations & Outlook
Small object detection performance remains suboptimal; reliance on large-scale pretrained models demands high computational resources; annotation quality may be constrained by data engine capabilities.
Plain Language Accessible to non-experts
Imagine searching for a specific book in a library. Traditional methods involve checking each book individually. WeDetect works like a smart library system that instantly retrieves books based on keywords. It’s fast and can handle complex queries like 'blue cover books.'
ELI14 Explained like you're 14
Think of playing a game where you need to find treasures on a map. Instead of searching every corner, WeDetect acts like a super helper that marks all treasure locations when you type 'gold treasure'! Cool, right?
Glossary
Open-Vocabulary Detection
Detects arbitrary categories using text prompts.
WeDetect enables large-scale category recognition.
Dual-Tower Architecture
Language and visual encoders operate independently, matching features in a shared embedding space.
WeDetect uses this architecture for efficient inference.
CLIP
A vision-language model aligning image-text pairs via contrastive learning.
WeDetect leverages CLIP pretraining.
Object Retrieval
Retrieves images containing user-specified objects.
WeDetect-Uni supports object retrieval tasks.
Referring Expression Comprehension
Locates objects based on complex language descriptions.
WeDetect-Ref handles complex expressions.
Open Questions Unanswered questions from this research
- 1 How to further improve small object detection performance?
- 2 How to optimize annotation quality from the data engine?
- 3 Cross-domain generalization needs deeper exploration.
Applications
Immediate Applications
Real-Time Object Detection
Applicable to surveillance, autonomous driving, and more, enabling fast multi-category recognition.
Content Moderation
Useful for social media platforms to automatically detect inappropriate content.
Long-term Vision
Intelligent Vision-Language Systems
Future applications in robotics and AR, supporting complex tasks.
Abstract
Open-vocabulary object detection aims to detect arbitrary classes via text prompts. Methods without cross-modal fusion layers (non-fusion) offer faster inference by treating recognition as a retrieval problem, \ie, matching regions to text queries in a shared embedding space. In this work, we fully explore this retrieval philosophy and demonstrate its unique advantages in efficiency and versatility through a model family named WeDetect: (1) State-of-the-art performance. WeDetect is a real-time detector with a dual-tower architecture. We show that, with well-curated data and full training, the non-fusion WeDetect surpasses other fusion models and establishes a strong open-vocabulary foundation. (2) Fast backtrack of historical data. WeDetect-Uni is a universal proposal generator based on WeDetect. We freeze the entire detector and only finetune an objectness prompt to retrieve generic object proposals across categories. Importantly, the proposal embeddings are class-specific and enable a new application, object retrieval, supporting retrieval objects in historical data. (3) Integration with LMMs for referring expression comprehension (REC). We further propose WeDetect-Ref, an LMM-based object classifier to handle complex referring expressions, which retrieves target objects from the proposal list extracted by WeDetect-Uni. It discards next-token prediction and classifies objects in a single forward pass. Together, the WeDetect family unifies detection, proposal generation, object retrieval, and REC under a coherent retrieval framework, achieving state-of-the-art performance across 15 benchmarks with high inference efficiency.