LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
LongEgoRefer benchmark challenges existing models with long-form egocentric video referring expression comprehension.
Key Findings
Methodology
The study introduces the LongEgoRefer benchmark for long-form egocentric video referring expression comprehension, leveraging the Ego4D dataset and EgoTracks annotations, using Gemini 2.5 Flash for expression generation, followed by human verification.
Key Results
- Existing models struggle with long-form videos; GPT-5 achieves best performance with 37.31% tIoU and 17.48% vIoU.
- Sparse target appearance in long videos challenges current models' localization capabilities.
- Two-stage pipeline shows potential but requires further optimization.
Significance
This research provides a new benchmark for long-form egocentric video referring expression comprehension, highlighting limitations of current models in long sequences, advancing video understanding.
Technical Contribution
Introduces the LongEgoRefer benchmark, combining automated and human-verified referring expressions, proposes a two-stage pipeline to tackle long-form video challenges.
Novelty
First to propose long-form egocentric video referring expression comprehension, filling the gap left by existing short-form benchmarks, offering a more challenging test environment.
Limitations
- Existing models perform poorly in long-form videos, especially with sparse target appearances.
- Two-stage pipeline suffers from error propagation issues.
Future Work
Future research can explore more effective long-form video understanding models, improve the two-stage pipeline, enhance temporal-spatial reasoning capabilities.
AI Executive Summary
Long-form egocentric video referring expression comprehension is a crucial task in video understanding, with existing benchmarks focusing on short clips, failing to reflect the complexity of real-world long videos.
The LongEgoRefer benchmark combines the Ego4D dataset and EgoTracks annotations, offering videos up to 45 minutes and detailed referring expressions, challenging existing models' spatio-temporal localization capabilities.
Experimental results show existing models perform poorly in long videos, especially with sparse target appearances, emphasizing the need for more robust video understanding models.
Deep Analysis
Background
Video referring expression comprehension is a critical task in computer vision, aiming to localize specific objects in videos based on natural language queries. Existing research focuses on short videos, failing to effectively handle complex scenes and sparse targets in long videos.
Core Problem
Sparse target appearances and complex human interactions in long-form egocentric videos make spatio-temporal localization highly challenging, with existing models struggling to effectively handle long sequences.
Innovation
The LongEgoRefer benchmark provides a task for referring expression comprehension in long videos, combining automated and human-verified expressions, proposing a two-stage pipeline to tackle long-form video challenges.
Methodology
- �� Dataset selection: Uses Ego4D long videos.
- �� Expression generation: Gemini 2.5 Flash automated generation and human verification.
- �� Two-stage pipeline: Combines VLM for temporal localization, Grounded SAM2 for spatial localization.
Experiments
Experiments use the Ego4D dataset, evaluating existing video REC models and VLMs in long videos, using metrics like tIoU and vIoU.
Results
GPT-5 performs best in long videos, but overall performance remains limited, with existing models struggling with sparse target appearances.
Applications
Long-form video referring expression comprehension can enhance AR and smart assistants, providing more natural human-machine interaction experiences.
Limitations & Outlook
Existing models' performance in long videos is limited by sparse target appearances and error propagation issues, requiring further optimization of model structures.
Plain Language Accessible to non-experts
Imagine you're shopping in a large supermarket, needing to find specific items. This task is like video referring expression comprehension: you have a shopping list (natural language query) and need to find items (target objects) in the supermarket (video). Long videos are like a huge supermarket, with items sparsely distributed, requiring patience and strategy to find.
ELI14 Explained like you're 14
Imagine you're playing a treasure hunt game, with a clue (natural language query), needing to find treasure (target object) on a huge map (video). This is like long-form video referring expression comprehension, finding sparsely appearing targets in a complex environment. It's challenging but also fun!
Glossary
Long-form video
Videos with extended durations, typically over 30 minutes.
Used to test models' spatio-temporal localization capabilities in long sequences.
Egocentric video
Videos shot from a first-person perspective, reflecting the camera wearer's subjective experience.
Used to study human-machine interaction and user intent.
Referring expression
Natural language query describing target objects in videos.
Used to test models' language-visual alignment capabilities.
Spatio-temporal grounding
Localizing target objects' time and space positions in videos.
Core task requiring integration of language and visual information.
Two-stage pipeline
A method processing tasks in stages, first temporal then spatial localization.
Used for referring expression comprehension in long videos.
Open Questions Unanswered questions from this research
- 1 How to improve models' spatio-temporal localization in long videos, especially with sparse targets?
- 2 How to optimize the two-stage pipeline to reduce error propagation?
Applications
Immediate Applications
Augmented Reality
Long-form video referring expression comprehension enhances AR devices for more natural user experiences.
Smart Assistants
Smart assistants can use long-form video understanding technology to better respond to user queries.
Long-term Vision
Human-Machine Interaction
Long-form video understanding technology can advance human-machine interaction, providing smarter interaction experiences.
Abstract
Egocentric videos capture rich and diverse human-object interactions and have emerged as a fundamental resource for understanding human activities related to objects. In this context, Video Referring Expression Comprehension (Video REC), the task of localizing the temporal and spatial extent of a referred object in video frames given a natural language query, plays a key role in linking textual descriptions to observed objects in untrimmed egocentric recordings. However, existing egocentric Video REC benchmarks primarily focus on short video clips, where some target object appears densely within frames. Such settings do not reflect real-world egocentric recordings, which are long-form, untrimmed, and characterized by sparse object occurrences and complex activity transitions. To address this limitation, we introduce LongEgoRefer, a novel and challenging benchmark constructed from long-form videos in the Ego4D dataset. LongEgoRefer contains 1,498 referring expressions with an average video duration of 45 minutes. The benchmark exhibits extreme target sparsity, detailed linguistic descriptions, and complex human-object interactions embedded in long, dynamic egocentric narratives. Consequently, it defines a demanding spatio-temporal grounding problem that requires models to identify both when an event occurs and where the referred object appears within extended video sequences. We evaluate existing Video REC approaches, including training-free baselines based on vision-language models combined with Grounded SAM2. Extensive experiments show that even advanced baselines and current state-of-the-art models struggle significantly on LongEgoRefer. These results highlight the intrinsic difficulty of long-form egocentric spatio-temporal grounding and emphasize the need for more robust video understanding models.