GptGet

ChatRex: Taming Multimodal LLM for Joint Perception and Understanding

TL;DR

ChatRex enhances multimodal LLM perception with decoupled design, achieving 72.8% recall on COCO dataset.

cs.CV 🔴 Advanced 2024-11-27 39 views
Qing Jiang Gen Luo Yuqin Yang Yuda Xiong Yihao Chen Zhaoyang Zeng Tianhe Ren Lei Zhang
AI Reader Arxiv Page Download PDF
multimodal large language model perception understanding dataset

Key Findings

Methodology

ChatRex employs a decoupled perception design using a Universal Proposal Network (UPN) to generate candidate boxes, which are then fed into a large language model (LLM), transforming the regression task into a retrieval task. On the data side, the Rexverse-2M dataset supports joint training of perception and understanding with multiple granularities.

Key Results

  • ChatRex achieves a recall rate of 72.8% on the COCO dataset, significantly outperforming Qwen2-VL's 43.9%.
  • On the RefCOCO/+/g datasets, ChatRex demonstrates strong performance in multimodal benchmarks, validating its combined perception and understanding capabilities.
  • Through a three-stage training strategy, ChatRex excels in both perception and understanding tasks, showcasing its potential in multimodal applications.

Significance

ChatRex addresses the limitations of existing multimodal large models in perception, particularly in object detection tasks. This research opens new possibilities for multimodal models in applications requiring precise perception, such as autonomous driving and robotic navigation.

Technical Contribution

By introducing the Universal Proposal Network (UPN) and Rexverse-2M dataset, ChatRex resolves modeling conflicts between perception and understanding tasks. Its innovative decoupled design enhances efficiency in handling perception tasks, demonstrating superior performance in multimodal tasks.

Novelty

ChatRex is the first to transform perception tasks into retrieval tasks through a decoupled design, resolving conflicts between perception and understanding. This innovation significantly enhances efficiency in handling multi-object detection.

Limitations

  • ChatRex may experience quantization errors when processing very large images, affecting perception accuracy.
  • The model's generalization ability in certain complex scenarios requires further validation.

Future Work

Future work could explore ChatRex's performance in more real-world applications, such as autonomous driving and robotic navigation. Further optimization of the model's perception accuracy and generalization ability is also a key research direction.

AI Executive Summary

ChatRex addresses the limitations of existing multimodal large models in perception. Current models like Qwen2-VL achieve only a 43.9% recall rate on the COCO dataset, limiting their application in tasks requiring precise perception. ChatRex, through its Universal Proposal Network (UPN) and Rexverse-2M dataset, transforms perception tasks into retrieval tasks, significantly enhancing perception capabilities. Experimental results show that ChatRex achieves a 72.8% recall rate on the COCO dataset and performs well on RefCOCO/+/g multimodal benchmarks. ChatRex's decoupled design not only enhances perception capabilities but also opens new possibilities for multimodal models in applications like autonomous driving and robotic navigation. However, ChatRex may experience quantization errors when processing very large images. Future work will continue to optimize the model's perception accuracy and generalization ability.

Deep Analysis

Background

Multimodal large models (MLLMs) excel in visual understanding but lack fine-grained perception capabilities. Current models like Qwen2-VL achieve only a 43.9% recall rate on the COCO dataset, limiting their application in tasks requiring precise perception. The modeling conflict between perception and understanding, and the lack of data supporting both, are major reasons.

Core Problem

Multimodal large models perform poorly in perception tasks, especially in object detection. Existing methods quantize bounding box coordinates into vocabulary tokens, which can lead to error propagation and ambiguity in prediction order, especially in multi-object detection.

Innovation

ChatRex transforms perception tasks into retrieval tasks using a decoupled design, employing a Universal Proposal Network (UPN) to generate candidate boxes that are fed into a large language model (LLM). This approach avoids quantization errors and ambiguity issues present in traditional methods.

Methodology

  • �� Use a Universal Proposal Network (UPN) to generate candidate boxes covering multiple granularities and categories.

  • �� Feed candidate boxes into a large language model (LLM), transforming perception tasks into retrieval tasks.

  • �� Construct the Rexverse-2M dataset to support joint training of perception and understanding with multiple granularities.

Experiments

Experiments were conducted on the COCO, LVIS, and RefCOCO/+/g datasets, using recall and precision as the main evaluation metrics. Through a three-stage training strategy, ChatRex excels in both perception and understanding tasks.

Results

ChatRex achieves a recall rate of 72.8% on the COCO dataset, significantly outperforming Qwen2-VL's 43.9%. On the RefCOCO/+/g datasets, ChatRex demonstrates strong performance in multimodal benchmarks, validating its combined perception and understanding capabilities.

Applications

ChatRex has potential in applications requiring precise perception, such as autonomous driving and robotic navigation. Its decoupled design enhances efficiency in handling multi-object detection.

Limitations & Outlook

ChatRex may experience quantization errors when processing very large images, affecting perception accuracy. Additionally, the model's generalization ability in certain complex scenarios requires further validation.

Plain Language Accessible to non-experts

Imagine you're shopping in a large supermarket with many products, and you need to find specific items. Traditional methods involve checking each shelf one by one, similar to how traditional multimodal large models handle perception tasks. ChatRex acts like a smart shopping assistant, marking all possible item locations first, allowing you to quickly find what you need based on these markers. This method not only improves efficiency but also reduces the likelihood of errors.

ELI14 Explained like you're 14

Imagine you're playing a treasure hunt game where you need to find all the treasures marked on a map. The traditional way is to find each treasure yourself, which can be slow and prone to mistakes. ChatRex acts like a super assistant, finding all possible treasure locations for you first, so you can just go to those spots to find the treasures. Isn't that cool?

Glossary

Multimodal Large Model (MLLM)

Models that combine multiple modalities (e.g., text, image) to understand and generate multimodal data.

Used to enhance visual understanding and perception capabilities.

Universal Proposal Network (UPN)

A network that generates candidate boxes covering multiple granularities and categories.

Used for perception tasks in ChatRex.

Rexverse-2M Dataset

A dataset supporting joint training of perception and understanding with multiple granularities.

Used for training ChatRex.

Decoupled Perception Design

A design that transforms perception tasks into retrieval tasks, avoiding quantization errors.

Core innovation of ChatRex.

Recall Rate

Measures the proportion of relevant samples correctly identified by the model.

Used to evaluate ChatRex's perception capabilities.

Open Questions Unanswered questions from this research

  • 1 How to further improve ChatRex's generalization ability in complex scenarios?
  • 2 How to reduce quantization errors in ChatRex when processing very large images?

Applications

Immediate Applications

Autonomous Driving

ChatRex can be used to enhance object detection capabilities in autonomous driving systems, especially in complex traffic scenarios.

Long-term Vision

Robotic Navigation

ChatRex can be applied to robotic navigation, helping robots more accurately identify and locate targets in complex environments.

Abstract

Perception and understanding are two pillars of computer vision. While multimodal large language models (MLLM) have demonstrated remarkable visual understanding capabilities, they arguably lack accurate perception abilities, e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on the COCO dataset, limiting many tasks requiring the combination of perception and understanding. In this work, we aim to bridge this perception gap from both model designing and data development perspectives. We first introduce ChatRex, an MLLM with a decoupled perception design. Instead of having the LLM directly predict box coordinates, we feed the output boxes from a universal proposal network into the LLM, allowing it to output the corresponding box indices to represent its detection results, turning the regression task into a retrieval-based task that LLM handles more proficiently. From the data perspective, we build a fully automated data engine and construct the Rexverse-2M dataset which possesses multiple granularities to support the joint training of perception and understanding. After a three-stage training approach, ChatRex demonstrates strong perception and understanding performance, and the combination of these two capabilities also unlocks many attractive applications, demonstrating their complementary roles in MLLM. Code is available at https://github.com/IDEA-Research/ChatRex.

cs.CV