UniPixel: Unified Object Referring and Segmentation for Pixel-Level Visual Reasoning
UniPixel unifies pixel-level perception and reasoning, achieving top performance on 10 benchmarks with 3B/7B models.
Key Findings
Methodology
Built on Qwen2.5-VL, UniPixel integrates a visual encoder, prompt encoder, object memory bank, and SAM 2.1 mask decoder. It processes multimodal inputs—images/videos, text, points/boxes/masks—and employs a multi-stage training strategy: pretraining on regional captioning, aligning segmentation with LLM via L→M projector, and fine-tuning with LoRA. The object memory bank stores dynamic spatial-temporal object info, enabling multi-round reasoning. The model outputs include spatial-temporal masks and textual responses, supporting complex pixel-level tasks.
Key Results
- On ReVOS, UniPixel 3B achieves 62.1 J&F, surpassing larger models by ~12%. On VideoRefer-BenchQ, it reaches 72.8% accuracy. Across 10 benchmarks, UniPixel outperforms SOTA, demonstrating robust multi-task pixel reasoning.
- In pixel-level referring and segmentation, UniPixel scores 74.9% IoU on RefCOCO+ and 70.8% on ReasonSeg, outperforming single-task models. The object memory enhances dynamic scene tracking.
- On the novel PixelQA task, UniPixel performs referring, segmentation, and QA simultaneously, showing strong generalization and task flexibility.
Significance
This work advances pixel-level multimodal understanding, bridging the gap between holistic perception and fine-grained reasoning. It addresses long-standing challenges in dynamic scene comprehension, enabling applications in autonomous driving, surveillance, and robotics. The unified framework sets a new standard for multi-task visual AI, inspiring future research in detailed scene understanding and interactive AI systems.
Technical Contribution
Introduces a unified multimodal framework with an object memory bank for dynamic spatial-temporal info, multi-stage training for cross-task generalization, and SAM 2.1 for effective mask decoding. The approach disentangles language and pixel prediction, enabling end-to-end pixel-level reasoning with multi-task outputs, outperforming existing methods in diverse benchmarks.
Novelty
First to propose an end-to-end model supporting simultaneous object referring, segmentation, and reasoning at pixel-level. The object memory bank uniquely maintains target consistency across frames, significantly improving dynamic scene tracking and multi-task fusion, setting a new paradigm in multimodal pixel-level understanding.
Limitations
- Performance drops under severe occlusion or cluttered backgrounds due to incomplete memory updates or ambiguous targets.
- High computational cost and training complexity limit real-time deployment, especially on high-resolution videos.
- Handling very long videos or extremely high-resolution scenes remains challenging, requiring further scalability and efficiency improvements.
Future Work
Future directions include developing more efficient memory mechanisms, reducing computational costs, and extending to real-time applications. Incorporating self-supervised learning could reduce reliance on annotated data. Expanding the framework to broader tasks like autonomous navigation and scene generation will further enhance its impact.
AI Executive Summary
Understanding complex scenes at the pixel level remains a fundamental challenge in computer vision. Existing large multimodal models excel at holistic tasks like captioning and question answering but struggle with fine-grained pixel-level reasoning, especially in dynamic scenes. Traditional approaches often treat referring and segmentation as separate tasks, limiting their ability to perform integrated visual reasoning.
To address this, the paper introduces UniPixel, a unified multimodal framework built upon the Qwen2.5-VL architecture. It combines a visual encoder, prompt encoder, object memory bank, and SAM 2.1 mask decoder to enable flexible, pixel-level understanding and reasoning. The object memory bank is a key innovation, storing spatial-temporal information about objects, allowing the model to maintain target consistency across frames and multi-round interactions.
The training strategy involves multiple stages: pretraining on regional captioning datasets, aligning segmentation with language models via the L→M projector, and fine-tuning with LoRA. This comprehensive approach ensures the model can handle diverse tasks, including referring, segmentation, and complex reasoning in videos. Experimental results on ten benchmarks demonstrate that UniPixel outperforms previous state-of-the-art methods, achieving a 62.1% J&F score on ReVOS with only 3B parameters, surpassing larger models by significant margins.
Furthermore, the model excels in the novel PixelQA task, which combines referring, segmentation, and question answering, validating its multi-task capabilities. Its ability to process multimodal inputs and generate detailed pixel-level outputs positions it as a breakthrough in fine-grained visual understanding. The research opens new avenues for applications in autonomous driving, robotics, surveillance, and interactive AI, where precise scene comprehension is crucial.
Looking ahead, future work will focus on improving efficiency, scalability, and real-time performance, as well as exploring self-supervised learning to reduce data dependency. Overall, UniPixel marks a significant step toward intelligent systems capable of detailed, dynamic scene understanding and reasoning at the pixel level.
Deep Analysis
Background
视觉理解技术经历了从目标检测到细粒度像素推理的演变。早期方法如Mask R-CNN实现了目标检测与分割，但多为静态场景。近年来，SAM、ViT等模型推动了整体理解，但在动态、多模态场景中仍有限。相关研究如LISA、VISA尝试融合指称与分割，但多支持单一任务，缺乏端到端统一架构。随着多模态交互需求增长，支持像素级感知、动态场景理解的模型成为研究热点。本文提出UniPixel，旨在突破现有局限，实现多任务、多模态的像素级理解，填补动态场景中的细粒度推理空白。
Core Problem
现有多模态模型多在整体理解或区域级别表现优异，但在像素级细粒度推理方面存在不足。主要问题包括：模型难以同时支持目标指称、像素分割和复杂推理，缺乏有效的目标空间-时间信息管理机制，以及在动态场景中的追踪和识别能力有限。这限制了其在自动驾驶、视频分析等应用中的部署。解决方案需设计支持多轮交互、动态记忆和多任务融合的统一架构，以实现更高精度和泛化能力的像素级理解。
Innovation
核心创新包括：1）引入对象记忆库，用于存储目标空间-时间信息，支持多轮交互中的目标一致性；2）多阶段训练策略，结合区域描述预训练、引导分割和微调，增强多任务能力；3）采用SAM 2.1作为掩码解码器，有效分离语言与像素预测能力；4）支持多模态输入（点、框、掩码）和多任务输出（指称、分割、问答），实现端到端统一推理。这些创新突破了单一任务模型的局限，为多模态像素级理解提供新思路。
Methodology
- �� 输入多模态信息（图像/视频、文本、点/框/掩码提示）
- �� 提示编码器将不同类型提示编码成高维向量
- �� 对象记忆库初始化为空，动态更新存储目标空间-时间信息
- �� 视觉编码器提取特征，掩码解码器生成空间-时间掩码
- �� 多阶段训练：区域描述预训练、引导分割对齐、LoRA微调
- �� 结合多任务损失（焦点、Dice、IoU、交叉熵）优化模型
- �� 支持多轮交互，利用记忆库增强目标一致性
- �� 输出文本响应和空间-时间掩码，实现像素级推理
Experiments
模型在ReVOS、VideoRefer-BenchQ、RefCOCO+、ReasonSeg等10项任务上评估。采用多阶段训练，使用大规模标注数据集，参数规模从3B到7B不等。对比SOTA模型，UniPixel在指标上均有明显提升，验证其多任务融合和像素级推理能力。Ablation研究显示对象记忆库显著提升动态场景中的目标追踪效果。模型还在新颖的PixelQA任务中表现出色，证明其跨任务泛化能力。
Results
UniPixel在ReVOS上，3B模型达62.1的J&F，超越现有7B模型约12%；在VideoRefer-BenchQ上达72.8%，优于对比模型。RefCOCO+和ReasonSeg上IoU分别达74.9%和70.8%。在PixelQA任务中，模型同时完成指称、分割和问答，展现强大多任务能力。这些结果验证了模型在复杂动态场景中的优越性能，推动像素级理解迈向新高度。
Applications
模型广泛应用于自动驾驶中的目标识别与追踪、视频监控中的异常检测、智能机器人中的场景理解，以及增强现实中的交互体验。其多模态输入和多任务输出设计，使系统适应多样化场景需求，提升智能水平。未来结合自监督学习，降低数据依赖，推动行业普及。
Limitations & Outlook
在极端遮挡或复杂背景下识别仍有限，主要因记忆库更新不足或目标模糊。训练成本高，推理速度有限，难以实时。对长视频或高分辨率场景适应性不足，需优化结构和效率。
Plain Language Accessible to non-experts
想象你在厨房做饭，手拿食材和工具。你需要记住每个食材的位置，还要根据菜谱一步步操作。这个模型就像一个聪明的厨师，不仅记住每个食材在哪，还能根据指示找到它们，做出美味菜肴。它可以在视频中找到特定物体，比如一只猫或一辆车，然后告诉你它们在什么位置，还能回答关于它们的问题。这个“厨师”用特别聪明的方式，把所有的食材和步骤记在脑海，确保每次都能准确完成任务。它还能追踪运动中的目标，就像在厨房里跟踪每个调料瓶一样。这样，它帮助你理解复杂场景，像个聪明的助手一样帮你解决问题。
ELI14 Explained like you're 14
想象你在玩一个超级酷的游戏，你可以用手指指着屏幕上的东西，比如一只狗或者一辆车，然后问：“那是什么？”或者“它在哪里？”这个模型就像那个会听懂你话的朋友，不仅能找到你指的东西，还能告诉你细节，比如颜色、位置，甚至还能讲故事。它还能看视频，追踪运动中的目标，比如一只跑来跑去的小猫。这“朋友”用特别聪明的方式，把你说的话和画面结合起来，帮你理解复杂场景。它就像一个会看会说的机器人助手，把视觉和语言结合，帮你解决各种难题。未来，它还能帮自动驾驶汽车识别路上的目标，或者帮机器人理解环境，变得更聪明、更贴心。
Abstract
Recent advances in Large Multi-modal Models (LMMs) have demonstrated their remarkable success as general-purpose multi-modal assistants, with particular focuses on holistic image- and video-language understanding. Conversely, less attention has been given to scaling fine-grained pixel-level understanding capabilities, where the models are expected to realize pixel-level alignment between visual signals and language semantics. Some previous studies have applied LMMs to related tasks such as region-level captioning and referring expression segmentation. However, these models are limited to performing either referring or segmentation tasks independently and fail to integrate these fine-grained perception capabilities into visual reasoning. To bridge this gap, we propose UniPixel, a large multi-modal model capable of flexibly comprehending visual prompt inputs and generating mask-grounded responses. Our model distinguishes itself by seamlessly integrating pixel-level perception with general visual understanding capabilities. Specifically, UniPixel processes visual prompts and generates relevant masks on demand, and performs subsequent reasoning conditioning on these intermediate pointers during inference, thereby enabling fine-grained pixel-level reasoning. The effectiveness of our approach has been verified on 10 benchmarks across a diverse set of tasks, including pixel-level referring/segmentation and object-centric understanding in images/videos. A novel PixelQA task that jointly requires referring, segmentation, and question answering is also designed to verify the flexibility of our method.