Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
Rex-Thinker uses Chain-of-Thought reasoning for object referring, enhancing precision and interpretability.
Key Findings
Methodology
Rex-Thinker employs Chain-of-Thought (CoT) reasoning for object referring. It first identifies all candidate object instances and then performs step-by-step reasoning to assess if they match the given expression. The method is trained using the HumanRef-CoT dataset, with a two-stage process involving cold-start supervised fine-tuning and GRPO-based reinforcement learning.
Key Results
- On the HumanRef benchmark, Rex-Thinker outperforms standard baselines in precision and interpretability, particularly in rejecting hallucinated outputs.
- In out-of-domain evaluation on the RefCOCOg dataset, Rex-Thinker demonstrates strong zero-shot generalization.
- GRPO fine-tuning further improves performance on RefCOCOg while maintaining reasoning capabilities across arbitrary object categories.
Significance
This study introduces Chain-of-Thought reasoning, significantly enhancing the interpretability and reliability of object referring tasks. It addresses the lack of verifiable reasoning processes and the difficulty in rejecting unmatched expressions in existing methods, offering a new perspective in visual understanding.
Technical Contribution
Rex-Thinker redefines the object referring task as a planning-action-summarization problem, providing a verifiable and trustworthy system. It introduces the HumanRef-CoT dataset to support interpretability training and uses GRPO reinforcement learning to enhance accuracy and generalization.
Novelty
Rex-Thinker is the first to apply Chain-of-Thought reasoning to object referring tasks, enhancing model interpretability and reliability through explicit reasoning processes, contrasting sharply with existing implicit methods.
Limitations
- In complex scenarios, the model may require longer reasoning times, potentially affecting real-time application performance.
- The model may face challenges when dealing with entirely novel object categories.
Future Work
Future research could explore optimizing reasoning speed and extending the model to handle more diverse object categories. Additionally, integrating other multimodal data sources may further enhance model performance.
AI Executive Summary
Rex-Thinker is a novel object referring model that achieves higher precision and interpretability through Chain-of-Thought reasoning. Traditional methods often predict bounding boxes directly, lacking interpretability and struggling to reject unmatched expressions. Rex-Thinker provides a verifiable reasoning process by evaluating each candidate object step-by-step to ensure it matches the given expression. Experimental results show that this method excels on the HumanRef benchmark and demonstrates strong zero-shot generalization on the RefCOCOg dataset. Although it may require longer reasoning times in complex scenarios, Rex-Thinker offers a new perspective and technical approach for object referring tasks.
Deep Analysis
Background
Object referring aims to identify all objects in an image that match a given natural language description. Traditional methods often treat it as a direct bounding box prediction task, lacking interpretability and reliability. Recently, multimodal large language models have made significant progress in this field, but they still face issues with opaque reasoning processes and hallucinated outputs.
Core Problem
The core problem with current object referring methods is the lack of interpretability and reliability. Models often fail to provide a clear reasoning process and struggle to reject unmatched expressions, limiting their reliability and usability in real-world applications.
Innovation
Rex-Thinker introduces Chain-of-Thought reasoning, redefining the object referring task as a planning-action-summarization problem. This method enhances model interpretability and reliability through explicit reasoning processes, contrasting sharply with existing implicit methods.
Methodology
- �� Use an open-vocabulary object detector to identify candidate objects
- �� Perform step-by-step reasoning to evaluate if each candidate matches the expression
- �� Train using HumanRef-CoT dataset with cold-start supervised fine-tuning
- �� Enhance accuracy and generalization through GRPO reinforcement learning
Experiments
Experiments were conducted using the HumanRef and RefCOCOg datasets, comparing Rex-Thinker with various baseline models. Key metrics included precision, recall, and interpretability. Ablation studies analyzed the contribution of each component.
Results
Rex-Thinker achieved the highest detection accuracy on the HumanRef benchmark and significantly reduced hallucinated outputs. On the RefCOCOg dataset, the model demonstrated strong zero-shot generalization, particularly in rejecting unmatched expressions.
Applications
Rex-Thinker can be applied to high-precision and interpretable visual understanding tasks, such as autonomous driving, intelligent surveillance, and human-computer interaction. Its reliable reasoning process makes it promising for safety-critical applications.
Limitations & Outlook
While Rex-Thinker excels in interpretability and reliability, it may require longer reasoning times in complex scenarios. Additionally, the model may face challenges when dealing with entirely novel object categories.
Plain Language Accessible to non-experts
Imagine you're looking for a specific toy. You first find all possible toys and then check each one to see if it matches what you want. That's how Rex-Thinker works. It uses step-by-step reasoning to ensure the object found truly matches the description, rather than just guessing. This approach not only improves accuracy but also makes each step visible, like carefully checking each toy in a toy store.
ELI14 Explained like you're 14
Imagine you're playing a game where the task is to find a person wearing a blue shirt. You'd first find all the people and then check each one's shirt color. That's how Rex-Thinker works! It uses step-by-step reasoning to make sure the object found truly matches the description, not just guessing. This way, the model acts like a smart detective, ensuring every choice is well-founded!
Glossary
Object Referring
Identifying objects in an image that match a given natural language description.
Rex-Thinker uses Chain-of-Thought reasoning for object referring.
Chain-of-Thought Reasoning
A step-by-step reasoning process ensuring each step is based on clear evidence.
Rex-Thinker employs Chain-of-Thought reasoning for object referring.
HumanRef-CoT
A large-scale dataset with step-by-step reasoning annotations.
Used to train Rex-Thinker to enhance its interpretability.
GRPO
A reinforcement learning method based on Group Relative Policy Optimization.
Used to enhance Rex-Thinker's accuracy and generalization.
Open-Vocabulary Object Detector
A detector capable of identifying objects across various categories.
Used to identify candidate objects in Rex-Thinker.
Open Questions Unanswered questions from this research
- 1 How to optimize reasoning speed in complex scenarios remains to be explored.
- 2 Potential for performance improvement when dealing with entirely novel object categories.
Applications
Immediate Applications
Intelligent Surveillance
Rex-Thinker can enhance target recognition accuracy in surveillance systems, especially in complex scenarios.
Long-term Vision
Autonomous Driving
With reliable object recognition and interpretability, Rex-Thinker can enhance the safety and decision-making capabilities of autonomous driving systems.
Abstract
Object referring aims to detect all objects in an image that match a given natural language description. We argue that a robust object referring model should be grounded, meaning its predictions should be both explainable and faithful to the visual content. Specifically, it should satisfy two key properties: 1) Verifiable, by producing interpretable reasoning that justifies its predictions and clearly links them to visual evidence; and 2) Trustworthy, by learning to abstain when no object in the image satisfies the given expression. However, most methods treat referring as a direct bounding box prediction task, offering limited interpretability and struggling to reject expressions with no matching object. In this work, we propose Rex-Thinker, a model that formulates object referring as an explicit CoT reasoning task. Given a referring expression, we first identify all candidate object instances corresponding to the referred object category. Rex-Thinker then performs step-by-step reasoning over each candidate to assess whether it matches the given expression, before making a final prediction. To support this paradigm, we construct a large-scale CoT-style referring dataset named HumanRef-CoT by prompting GPT-4o on the HumanRef dataset. Each reasoning trace follows a structured planning, action, and summarization format, enabling the model to learn decomposed, interpretable reasoning over object candidates. We then train Rex-Thinker in two stages: a cold-start supervised fine-tuning phase to teach the model how to perform structured reasoning, followed by GRPO-based RL learning to improve accuracy and generalization. Experiments show that our approach outperforms standard baselines in both precision and interpretability on in-domain evaluation, while also demonstrating improved ability to reject hallucinated outputs and strong generalization in out-of-domain settings.