I Seek You in Videos: Identity-Conditioned Queries for Person-Centric Video Reasoning
Proposes ICQ task, builds ISYV dataset and model, achieving 57% accuracy on person-centric video reasoning, improving long-horizon and cross-domain identification.
Key Findings
Methodology
This work introduces the Identity-Conditioned Query (ICQ) task, which requires models to process both a complex video and a reference image of a target person. The core component, ICQ module, compresses reference images into learnable tokens, facilitating effective multimodal feature fusion via attention mechanisms. The training employs a two-stage pipeline: supervised fine-tuning (SFT) followed by reinforcement fine-tuning (RFT) using the Group Relative Policy Optimization (GRPO) algorithm, enhanced with a novel ESR reward that automatically selects relevant evidence without additional annotations. The dataset construction involved automated annotation, multi-stage verification, and manual review, resulting in the ISYV-75K training set with 75,000 samples and the ISYV-Bench evaluation benchmark with 1377 videos spanning six hierarchical cognitive levels. Experiments demonstrate that current state-of-the-art models perform poorly on this benchmark, with average accuracy below 40%, whereas the proposed ISYV-Model reaches 57%, approaching closed-source performance.
Key Results
- On ISYV-Bench, mainstream models achieve below 40% accuracy, while our ISYV-Model with 7B parameters reaches 57%, showing significant improvement especially in long-term tracking and cross-scene recognition.
- Ablation studies confirm that the ICQ module and ESR-based reinforcement learning substantially enhance identity matching and reasoning capabilities, with improvements of over 15% in cross-domain scenarios.
- The model demonstrates robust performance in complex scenes involving costume changes, occlusions, and multi-person interactions, validating its effectiveness in real-world applications.
Significance
This research advances the field of multimodal video understanding by integrating reference images into long-form video reasoning, enabling models to perform fine-grained identity tracking and causal inference across diverse scenarios. It addresses key limitations of existing benchmarks that focus solely on video-text interactions, providing a more comprehensive evaluation aligned with human cognitive development. The datasets and models developed here serve as valuable resources for future research, pushing the boundaries of AI's perceptual and reasoning capabilities in complex, real-world environments. The ability to perform accurate long-horizon tracking and behavior understanding has profound implications for surveillance, personalized content retrieval, and intelligent video editing, bridging the gap between AI perception and human-like cognition.
Technical Contribution
The paper introduces the ICQ task framework, which extends traditional video reasoning by incorporating a reference image, and designs a dedicated ICQ module that compresses this image into a set of learnable tokens for efficient multimodal fusion. The training strategy combines supervised fine-tuning with reinforcement learning based on the GRPO algorithm, utilizing a novel ESR reward to automatically identify relevant evidence without shot-level annotations. The datasets include a large-scale 75K training set (ISYV-75K) and a challenging evaluation benchmark (ISYV-Bench), both designed to cover diverse scenarios and cognitive levels. The model architecture leverages multi-layer attention mechanisms and hierarchical task design, enabling it to handle complex reasoning tasks such as social dynamics, spatial memory, and causal inference. Experimental results demonstrate the model's superiority over existing baselines, especially in cross-domain and long-term scenarios.
Novelty
This work is the first to formalize the ICQ task, integrating video, reference images, and textual queries into a unified person-centric reasoning framework. Its key innovations include the ICQ module for compact reference image representation, a hierarchical cognitive task design inspired by human development, and a novel reinforcement learning reward that automatically selects relevant evidence. Unlike prior works limited to static images or simple video-text tasks, this approach emphasizes cross-domain, long-horizon reasoning with multi-source inputs. The datasets created are also unprecedented in scale and complexity, enabling comprehensive evaluation of models' cognitive and perceptual abilities in person-centric scenarios.
Limitations
- The model's robustness diminishes under extreme lighting, occlusion, or background clutter, primarily due to limitations in visual feature extraction robustness.
- Cross-domain generalization remains challenging, especially when encountering novel scenes with minimal training samples, indicating a need for better domain adaptation techniques.
- Training large-scale models with multi-source data incurs high computational costs, limiting accessibility and scalability for broader deployment.
Future Work
Future directions include developing more efficient model compression techniques to reduce computational costs, enhancing robustness against visual variations, and expanding multi-modal inputs to include audio and motion cues. Additionally, extending ICQ to multi-target scenarios and integrating causal reasoning for more sophisticated behavior understanding are promising avenues. Further research is needed to improve cross-domain adaptability and real-time inference capabilities, aiming to deploy these models in practical surveillance, autonomous systems, and interactive media.
AI Executive Summary
In recent years, the rapid development of multimodal large language models (MLLMs) has revolutionized AI's capacity to understand complex visual and textual data. However, most existing video reasoning benchmarks focus narrowly on tasks like captioning, question answering, or grounding, primarily relying on paired video-text inputs. These approaches, while effective in controlled settings, fall short when faced with real-world scenarios requiring nuanced, person-centric understanding across diverse sources and long temporal horizons.
This gap motivated the authors to propose a novel task—Identity-Conditioned Query (ICQ)—which integrates a target person's reference image with a video, challenging models to perform identity grounding, behavior analysis, and causal reasoning simultaneously. The core idea is to enable models to process heterogeneous inputs—complex videos with multiple scenes and a reference image—and leverage this information to track and interpret the target individual across various contexts.
To operationalize this task, the researchers developed the ISYV system, comprising three main components. First, they curated the ISYV-Bench, a challenging evaluation dataset with 1377 videos featuring complex shot transitions, diverse scenes, and multiple identities, organized into six hierarchical cognitive levels inspired by human development. Second, they assembled the ISYV-75K dataset with 75,000 high-quality samples, generated through automated annotation, multi-stage verification, and manual review, ensuring diversity and realism. Third, they designed the ISYV-Framework, a model architecture that incorporates an ICQ module—compressing reference images into learnable tokens—and employs a two-stage training strategy combining supervised fine-tuning with reinforcement learning guided by a novel ESR reward.
Extensive experiments reveal that current mainstream models perform poorly on this benchmark, with accuracy below 40%. In contrast, the proposed ISYV-Model achieves 57% accuracy at 7B scale, demonstrating significant progress in long-term, cross-scene person tracking and reasoning. Ablation studies confirm the effectiveness of the ICQ module and the reinforcement learning approach, highlighting their roles in improving identity matching and causal inference.
This work marks a substantial step forward in person-centric video understanding, addressing key challenges of heterogeneous input integration, long-horizon reasoning, and cross-domain generalization. Its datasets and methodologies provide valuable resources for future research, aiming to develop AI systems capable of human-like perception and reasoning in complex, real-world environments. Future efforts will focus on reducing computational costs, enhancing robustness, and expanding multi-modal capabilities, paving the way for smarter surveillance, personalized content retrieval, and interactive AI applications.
Deep Dive
Abstract
Real-world video reasoning often involves multimodal, multi-source inputs, whereas existing video reasoning tasks typically assume a simplified video-text setting, limiting identity matching and person-centric reasoning. To bridge this gap, we introduce the Identity-conditioned Queries (ICQ) task, in which models are required to jointly associate and interpret an input video and a reference image of a person, and leverage this conditioning to address identity grounding, behavior understanding, and temporal reasoning, among other challenges. Building on ICQ, we present ISYV (I Seek You in Videos), a systematic solution comprising three components: (1) ISYV-Bench, a challenging evaluation benchmark with 1,377 real-world complex videos and 1,377 question-answer pairs, organized into six difficulty levels spanning capabilities from identity recognition to causal reasoning; (2) ISYV-75K, a large-scale training set of 75K high-quality samples constructed via automated annotation, multi-stage verification, and manual review; and (3) ISYV-Framework, containing an ICQ-oriented model and training strategy for learning to exploit informative video shots without additional shot-level annotations. Extensive experiments show that both mainstream closed-source and open-source MLLMs struggle on ISYV-Bench, especially in cross-domain identity matching and long-horizon tracking. ISYV-Model outperforms strong baselines and in some aspects approaches closed-source performance. Overall, ISYV provides a unified task definition, scalable datasets/benchmarks, and modeling insights for person-centric video reasoning.
References (20)
Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities
Gheorghe Comanici, E. Bieber, Mike Schaekermann et al.
Qwen3 Technical Report
An Yang, Anfeng Li, Baosong Yang et al.
Multi-GRPO: Multi-Group Advantage Estimation for Text-to-Image Generation with Tree-Based Trajectories and Multiple Rewards
Qian Lyu, Zicong Chen, Chongxiao Wang et al.
A Stable Long-Term Tracking Method for Group-Housed Pigs
Shibo Gao, Jinmeng Gong, Peipei Yang et al.
GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learning
GLM-V Team Wenyi Hong, Wenmeng Yu, Xiaotao Gu et al.
Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
Haoji Zhang, Xin Gu, Jiawen Li et al.
InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
Jinguo Zhu, Weiyun Wang, Zhe Chen et al.
VCapsBench: A Large-scale Fine-grained Benchmark for Video Caption Quality Evaluation
Shi-Xue Zhang, Hongfa Wang, Duojun Huang et al.
ReaSon: Reinforced Causal Search with Information Bottleneck for Video Understanding
Yuan Zhou, Litao Hua, Shilong Jin et al.
Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers
Zhao-yu Su, Peng Xia, Hangyu Guo et al.
VAGU & GtS: LLM-Based Benchmark and Framework for Joint Video Anomaly Grounding and Understanding
Shibo Gao, Peipei Yang, Yangyang Liu et al.
EgoExoLearn: A Dataset for Bridging Asynchronous Ego- and Exo-centric View of Procedural Activities in Real World
Yifei Huang, Guo Chen, Jilan Xu et al.
Facial Geometric Detail Recovery via Implicit Representation
Xingyu Ren, Alexandros Lattas, Baris Gecer et al.
TVQA: Localized, Compositional Video Question Answering
Jie Lei, Licheng Yu, Mohit Bansal et al.
TALL: Temporal Activity Localization via Language Query
J. Gao, Chen Sun, Zhenheng Yang et al.
MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
Kunchang Li, Yali Wang, Yinan He et al.
Achieving binary weight and activation for LLMs using Post-Training Quantization
Siqing Song, Chuang Wang, Ruiqi Wang et al.
Qwen2.5-VL Technical Report
Shuai Bai, Ke-qin Chen, Xuejing Liu et al.
The Development of Cognitive Anthropology
R. D'Andrade
DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
Jinyoung Park, Jeehye Na, Jinyoung Kim et al.