Homer: Understanding Long-form Videos with Hierarchical Memory and Agentic Reasoning
HOMER employs hierarchical memory and agentic reasoning, achieving +10.8 performance on long video benchmarks.
Key Findings
Methodology
HOMER constructs a three-layer hierarchical memory: perceptual buffer for keyframes, entity graph for character memories, and event graph for causal event links. An agentic reasoner performs multi-round retrieval, verification, and correction, decomposing questions into subtasks. Techniques include incremental event graph building, entity matching, and multimodal retrieval strategies, balancing coverage and efficiency. The system integrates explicit causal links and multi-hop reasoning, enabling structured understanding of long videos.
Key Results
- HOMER outperforms previous methods by +5.5, +10.8, and +4.4 points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long, respectively. Its model-agnostic design demonstrates broad applicability across different backbone models. Ablation studies confirm the importance of causal event links and multi-round retrieval, significantly improving reasoning accuracy in complex scenarios.
- The multi-level retrieval and verification mechanisms enhance evidence localization and reasoning robustness. The hierarchical memory effectively organizes information, reducing missing details and improving multi-hop inference. Experiments validate that explicit causal links and iterative validation are key to performance gains, especially in long, dense videos.
- Across diverse datasets, HOMER maintains consistent improvements, showing its scalability and generality. Its structure addresses core challenges like information dispersion, temporal complexity, and causal reasoning, setting new standards for online long video understanding.
Significance
This work advances long video understanding by integrating explicit causal modeling with hierarchical memory and active reasoning. It addresses longstanding issues of information fragmentation and reasoning depth, enabling more accurate and interpretable AI systems for applications like autonomous driving, content analysis, and surveillance. Its model-agnostic design ensures broad utility, fostering future research in structured temporal reasoning and multimodal integration. The framework paves the way for AI that can comprehend extended narratives with human-like coherence and depth.
Technical Contribution
HOMER introduces a multi-layer hierarchical memory system combined with an agentic reasoning loop that performs multi-round, verified retrievals. Its explicit causal event graph captures long-range dependencies, while the verification and self-evolution mechanisms improve reasoning reliability and adaptability. The approach is model-agnostic, compatible with various large language models, and emphasizes structured, multi-hop evidence integration, representing a significant step forward in online long video understanding.
Novelty
This is the first work to combine explicit hierarchical memory with an active, verification-based reasoning process tailored for online long video comprehension. Unlike prior offline or coarse online methods, HOMER models detailed causal relationships between events, enabling multi-hop reasoning in streaming scenarios. Its model-agnostic, multi-layered design offers a new paradigm for structured, scalable long video AI, filling a critical gap in current methodologies.
Limitations
- The model struggles with extremely long videos containing sparse or highly complex causal chains, as the explicit graph may become fragmented or incomplete. Computational costs of multi-round retrieval can be high, limiting real-time deployment in resource-constrained environments.
- Robustness to noisy or ambiguous multimodal data remains a challenge, especially when modality-specific signals are weak or conflicting. Further improvements are needed for broader applicability in diverse real-world scenarios.
- While the hierarchical structure improves organization, it relies on predefined event segmentation and causal annotations, which may not generalize well to unstructured or novel content. Future work should explore adaptive, data-driven event modeling.
Future Work
Future directions include integrating self-supervised learning to enhance event and entity representations, optimizing retrieval strategies for efficiency, and extending causal modeling to handle more complex, unstructured narratives. Incorporating additional modalities like audio-visual cues or sensor data could further improve understanding. Developing more scalable architectures and real-time inference capabilities will be crucial for practical deployment in autonomous systems and content analysis.
AI Executive Summary
Understanding long videos poses a fundamental challenge in artificial intelligence, especially when aiming for human-like comprehension of extended narratives. Existing approaches often rely on sparse sampling, compression, or offline processing, which struggle to balance information retention and reasoning depth. These methods typically lack explicit modeling of causal relationships and multi-hop reasoning, limiting their effectiveness in complex scenarios.
HOMER introduces a novel framework inspired by human cognition, employing a hierarchical memory system that organizes information at multiple levels: raw perception, character-centric memories, and event-level causal links. This structure allows the system to efficiently store and retrieve relevant information, even across long temporal spans. The core innovation lies in the agentic reasoner, which decomposes questions into subtasks, performs multi-round retrievals, and verifies each step through a validation mechanism. This process ensures accurate, coherent answers while dynamically recovering from retrieval errors.
Experimental results demonstrate HOMER’s superiority across multiple benchmarks, including M3-Bench and Video-MME-Long, with performance gains of up to +10.8 points. The model’s design is model-agnostic, compatible with various large language backbones, and significantly enhances multi-hop reasoning capabilities. Its explicit causal event graph captures long-range dependencies, addressing the core challenge of information dispersion in long videos.
This work has broad implications for AI applications such as autonomous driving, video content analysis, and intelligent surveillance, where understanding extended temporal contexts is critical. Despite its strengths, challenges remain in scaling to extremely long or noisy videos and reducing computational costs. Future research will focus on adaptive event modeling, multimodal integration, and efficiency improvements, aiming to bring AI closer to human-level long-term understanding.
Deep Analysis
Background
长视频理解是多模态人工智能的核心问题之一，早期方法如VideoTree和LongVLM通过稀疏抽样或区域级特征提取，试图在有限模型容量内捕获关键信息。近年来，主动推理模型如VideoAgent和LongVT引入多轮交互，增强证据整合能力，但多为离线场景，难以满足实时在线需求。在线方法如Wang et al.的稀疏记忆和Zeng et al.的事件森林，试图在有限记忆中组织信息，但缺乏显式事件因果关系建模，导致多跳推理困难。整体而言，长视频理解仍面临信息碎片化、因果关系复杂和推理深度不足的挑战。
Core Problem
核心问题在于如何在有限记忆空间中，有效组织长视频中的信息，特别是显式建模事件间的因果关系，实现多跳推理。现有在线方法多采用粗粒度记忆或单一检索机制，难以兼顾信息覆盖和细节保留，导致推理准确率不足。此外，缺乏主动推理和验证机制，容易受到噪声干扰，难以应对复杂场景。这些问题限制了长视频理解的深度和广度，亟需创新的结构和机制。
Innovation
HOMER的创新在于：1）引入多层次层级记忆体系，将感知、人物、事件分层存储，显式建模事件的时间和因果关系；2）采用主动推理机制，通过多轮检索、验证和修正，动态定位证据，提升多跳推理能力；3）模型无关设计，兼容多种基础大模型，具有良好的扩展性；4）引入任务调度与验证机制，确保推理步骤的正确性。这些创新突破了传统单一记忆或离线模型的限制，为在线长视频理解提供了新思路。
Methodology
- �� 构建三层记忆体系：感知缓冲区存储关键帧，实体图组织人物记忆，事件图链接事件段的因果关系。
- �� 采用增量构建策略：事件图在视频流中动态创建、扩展和连接，确保结构随视频演进而优化。
- �� 多轮主动推理：通过任务分解，将问题拆解为子任务，利用多模态检索（事件、实体、关键帧）逐步收集证据。
- �� 验证-修正机制：每轮检索后，模型验证证据充分性，必要时进行修正或重检。
- �� 任务调度：维护任务清单，优先处理未解决子任务，保证推理连续性。
- �� 自我演化：总结推理经验，优化检索策略，提升模型自主学习能力。
Experiments
在M3-Bench-robot、M3-Bench-web和Video-MME-Long三个数据集上，采用问答和多项选择评估指标，比较HOMER与SOTA方法。设置不同模型基础（如GPT-3.5、Llama-2）验证模型无关性。通过消融实验，验证事件图因果连接、多轮检索和验证机制的重要性。参数调优包括记忆层数、检索轮次、验证阈值等，确保模型在长视频中的高效表现。评估指标涵盖准确率、召回率和推理深度。
Results
HOMER在三个数据集上均实现性能突破，分别超越最优方法5.5、10.8和4.4分。多轮检索和验证机制显著提升多跳推理准确率，验证了模型在复杂场景中的鲁棒性。层级记忆结构有效组织信息，减少了信息遗漏。模型兼容多种基础模型，表现出良好的扩展性和适应性。消融实验确认事件因果连接和多轮验证是性能提升的核心因素。
Applications
该框架适用于智能视频内容分析、自动驾驶场景理解、智能监控等领域。只需输入长视频流，模型即可实现实时结构化理解和推理，支持多模态信息融合。未来可结合边缘计算，推动智能监控和自动驾驶的自主决策能力提升。还可应用于影视内容分析、虚拟助手等多种场景，极大丰富人机交互的智能化水平。
Limitations & Outlook
当前模型对极端复杂或噪声较多的场景仍存在理解偏差，因其依赖显式关系和预定义结构。多轮检索带来较高计算成本，不适合超大规模实时应用。模型在多模态信息融合方面仍需优化，面对模态缺失或不一致时表现不足。未来需提升鲁棒性和效率，扩展模型的适应范围。
Plain Language Accessible to non-experts
想象你在整理一份长长的家庭相册，每一张照片都记录了不同的时刻。有些照片是家庭聚会，有些是旅行风景，还有一些是重要的事件。要理解整个家庭故事，你需要把这些照片按照时间顺序和事件关系整理起来，找出谁在什么场合做了什么事。HOMER就像这个整理者，它用不同的“文件夹”存放不同类型的记忆：一层存照片，一层是关于人物的记忆，另一层是连接事件的故事线。每当有人问你一个关于家庭的故事，它会先找到相关的照片，然后查查人物的细节，最后把故事拼凑完整。它还会反复检查，确保每个细节都准确无误。这样，长视频中的复杂故事就变得像整理家庭相册一样清晰有序，容易理解。
ELI14 Explained like you're 14
想象你在看一部很长的电影，里面有很多不同的场景、人物和故事线。要理解整部电影，你不能只看一眼就知道全部内容，而是要记住每个重要的场景、人物的动作和他们之间的关系。HOMER就像一个聪明的助手，它把电影的内容分成三个部分：一部分是关键画面，帮助你快速回忆；一部分是关于人物的记忆，告诉你谁是谁、他们做了什么；还有一部分是故事的线索，把不同场景连接起来，告诉你事件是怎么发生的。每当你问它一个问题，它会先找到相关的场景，然后查查人物的细节，最后把答案拼凑出来。它还会反复确认信息的正确性，确保你得到的答案是准确的。这样一来，即使电影很长，也能像看短片一样明白整个故事。
Glossary
Hierarchical Memory (层级记忆)
一种多层次组织信息的结构，从感知到事件关系逐层递进，便于长时序推理。
HOMER的三层记忆体系，分别是感知缓冲、实体图和事件图。
Agentic Reasoner (主动推理器)
一种具有主动检索、验证和修正能力的推理机制，模拟人类主动寻找证据的过程。
核心在于多轮检索与验证，提升长视频多跳推理能力。
EventGraph (事件图)
表示事件段之间的时间和因果关系的有向图，用于捕捉长视频中的叙事结构。
支持多跳推理和事件关系建模。
Multi-round Retrieval (多轮检索)
多次交互式检索证据，每轮根据前一轮结果调整搜索策略。
提升复杂推理的准确性。
Causal Links (因果关系)
事件之间的因果连接，描述事件的因果依赖关系。
显式建模事件的因果链，增强推理深度。
Open Questions Unanswered questions from this research
- 1 如何在极端长视频中保持高效推理，减少计算成本仍是挑战。未来需要探索更智能的记忆压缩和推理优化策略。
- 2 多模态信息融合的鲁棒性不足，面对模态缺失或噪声时模型表现有限，需进一步提升模型的适应能力。
Abstract
Multimodal large language models excel on short clips but struggle on hour-long videos in an online setting, where frames are processed incrementally under limited memory. Existing online methods either retain compact visual representations that lack semantic structure, or build higher-level memory stores organized around temporal proximity rather than explicit causal links, leaving multi-hop narrative reasoning to be reconstructed by the LLM at every query. We bridge this gap with \textsc{Homer}, a Hierarchical Online Memory Exploration and Reasoning framework. \textsc{Homer}'s memory mirrors the multi-scale structure of long videos, ranging from raw perception, to recurring entities, to events connected by explicit temporal and causal relations. Its agentic reasoner then explores this memory the way humans do, locating the relevant scene, looking up details, and composing the answer through multi-round memory retrieval, with a harness that verifies and corrects each step. \textsc{Homer} outperforms the previous best agent method by $+5.5$, $+10.8$, and $+4.4$ points on M3-Bench-robot, M3-Bench-web, and Video-MME-Long, and consistently lifts three various LLM backbones, indicating a model-agnostic structural capability for grounded retrieval over long videos.