FOLIO: Focused Semantic Memory for Streaming Video Understanding
FOLIO is a training-free focused semantic memory system combining short-term visual buffers with long-term entity-centric structured memory, boosting streaming video understanding.
Key Findings
Methodology
FOLIO employs a segment-wise online memory update framework guided by a dynamic focus state. It integrates a short-term visual buffer, a long-term entity-centered structured memory, and a visual evidence cache. Keyframes are selected per segment, with focus scores prioritizing entities and actions for detailed storage. The system uses rule-based memory writing, combining structured record generation via a writer VLM with focus-driven update rules. During retrieval, lightweight matching and semantic expansion enable efficient multi-turn querying. This approach avoids training, relying on feature-driven rules and explicit entity management, effectively balancing memory cost and information richness.
Key Results
- On OVO-Bench, FOLIO achieves 82.0/69.1 perception/backward accuracy, outperforming prior methods significantly, and reaches 74.5% overall accuracy on StreamingBench. Its focus-guided memory prioritization reduces redundant storage while maintaining high long-horizon understanding. Multi-turn experiments show that updating focus states with previous query context enhances entity tracking and long-term reasoning, with semantic expansion improving retrieval accuracy by approximately 15% over pure matching.
- Ablation studies confirm that the combination of focus-guided writing and structured memory is critical for performance gains. The system demonstrates robustness across diverse scenarios, maintaining low memory overhead while achieving state-of-the-art accuracy, especially on tasks requiring long-term grounding and multi-entity tracking.
- The results validate that selective, focus-driven memory organization effectively addresses the challenges of streaming video comprehension, enabling scalable, real-time understanding with reduced computational and storage costs.
Significance
This work addresses fundamental bottlenecks in streaming video understanding by proposing a novel, training-free semantic memory architecture that dynamically prioritizes important entities and actions. It overcomes the limitations of traditional frame-based or event-based memory systems, which often suffer from redundancy, inefficient long-term tracking, and high computational costs. By integrating a focus mechanism with structured entity memory, FOLIO facilitates scalable, accurate, and real-time scene comprehension, applicable to autonomous vehicles, surveillance, and interactive AI agents. Its ability to balance detailed memory with compact context storage marks a significant step toward practical, deployable streaming perception systems. Future directions include multi-modal fusion, adaptive focus mechanisms, and reinforcement learning-based memory optimization.
Technical Contribution
FOLIO introduces a hybrid, rule-driven memory update framework that combines short-term visual buffers with long-term entity-centric structured memory, guided by a dynamic focus state. It departs from prior frame, event, or cache-based methods by explicitly modeling entity identities, relations, and actions, enabling long-horizon tracking. The focus state dynamically adjusts the storage priority based on interaction signals, visual changes, and previous query context. Its lightweight hybrid retrieval combines direct structured matching with semantic expansion, supporting multi-turn reasoning. These innovations collectively enable scalable, accurate, and efficient streaming video understanding without requiring training, setting a new paradigm for online scene comprehension.
Novelty
This is the first work to propose a training-free, focus-guided semantic memory system that organizes information around observed entities with dynamic prioritization. Unlike prior methods relying solely on frame caches or event hierarchies, FOLIO’s entity-centered structured memory and focus mechanism allow for selective, long-term tracking of targets over unbounded streams. Its hybrid retrieval strategy further distinguishes it from existing approaches, enabling efficient multi-turn reasoning with minimal overhead. This combination of explicit entity management, focus-driven updates, and semantic retrieval represents a novel contribution to streaming video understanding.
Limitations
- The reliance on rule-based focus and memory update mechanisms may limit adaptability in highly dynamic or cluttered scenes, where automatic focus adjustment could falter. The system’s performance depends on the quality of keyframe selection and feature features, which may degrade under severe occlusion or rapid scene changes. Additionally, the current architecture may face scalability issues with extremely long sequences or dense multi-entity scenarios, requiring further optimization or learning-based focus adaptation.
- While avoiding training, the system’s rule heuristics may not generalize well across diverse domains without manual tuning. Its computational efficiency, though improved, still faces challenges in real-time deployment in resource-constrained environments. Future work should explore adaptive, learning-based focus mechanisms and multi-modal integration to enhance robustness and scalability.
Future Work
Future research will focus on integrating reinforcement learning to adaptively tune focus priorities based on scene context and task demands. Multi-modal fusion, incorporating audio, text, and sensor data, will be explored to enrich entity representations and improve reasoning. Additionally, scalable architectures and hardware acceleration strategies are needed to deploy FOLIO in real-time, resource-limited scenarios such as autonomous vehicles and edge devices. Extending the framework to unsupervised or weakly supervised settings could further reduce reliance on manual rule design, enabling broader applicability.
AI Executive Summary
Understanding streaming videos in real time remains a significant challenge due to the unbounded growth of visual data and the need for efficient, long-term memory management. Traditional methods often rely on storing raw frames or event summaries, which become computationally expensive and redundant over time. This paper introduces FOLIO, a novel, training-free semantic memory system that addresses these issues by combining a short-term visual buffer with a long-term, entity-centered structured memory guided by a dynamic focus state.
FOLIO’s core innovation lies in its focus-guided memory update mechanism. As each video segment arrives, the system selects keyframes based on visual change and interaction signals, then generates structured records for important entities and actions using a writer VLM. These records are merged into a persistent memory chain, which maintains entity identities, states, locations, and relations over time. The focus state dynamically adjusts the storage priority, ensuring that critical targets receive richer, more detailed memory updates, while background information remains compact.
At query time, FOLIO employs a lightweight hybrid retrieval strategy. It first performs direct matching over the structured memory, then applies semantic expansion via a VLM-assisted process to retrieve relevant evidence efficiently. This approach supports multi-turn interactions, where previous query results update the focus state, enabling continuous long-term tracking and reasoning. Extensive experiments on OVO-Bench and StreamingBench demonstrate that FOLIO surpasses state-of-the-art performance, achieving 82.0/69.1 accuracy and 74.5% overall, respectively, while significantly reducing memory maintenance costs.
The significance of this work is profound: it offers a scalable, accurate solution for real-time streaming video understanding, applicable to autonomous driving, surveillance, and intelligent agents. Its ability to organize long-term semantic memory without training paves the way for more adaptive, resource-efficient systems. Future directions include multi-modal integration, reinforcement learning for focus adaptation, and deployment in real-world scenarios, promising a new era of continuous scene comprehension.
Deep Analysis
Background
随着视频内容的不断丰富与复杂化，流媒体视频理解逐渐成为人工智能研究的热点。早期方法多依赖离线分析或全视频访问，难以满足实时性需求。近年来，诸如Video-LLMs、层次记忆和事件驱动存储等技术不断发展，解决了部分长时依赖和多目标追踪问题，但仍面临存储成本高、信息冗余和多轮交互效率低等难题。实际应用中，系统需在有限资源下持续处理无限增长的视觉信息，保持高精度和低延迟，成为研究难点。
Core Problem
核心问题在于如何在视频流不断增长的情况下，有效组织和存储关键信息，避免冗余，同时支持多轮复杂查询。传统方法多采用帧缓存或事件存储，难以兼顾信息丰富性和存储效率。长时依赖追踪和多目标识别在高动态场景中尤为困难，特别是在有限计算资源下实现实时响应。解决方案需创新存储策略和高效检索机制，以实现长时记忆的有选择性存储和快速访问。
Innovation
本研究的创新点包括：1）提出无训练的焦点引导语义记忆，通过动态焦点状态优先存储重要实体和动作，提升存储效率；2）结合短期视觉缓冲与长时结构化记忆，兼顾实时性与长时依赖；3）采用规则驱动的写入策略，避免无差别存储带来的成本增加；4）引入轻量级的结构化检索和语义扩展，提升多轮交互中的目标追踪能力。这些创新共同解决了长时依赖和多目标管理难题，为流媒体视频理解提供新思路。
Methodology
- �� 视频按段划分，逐段处理；
- �� 每段选择关键帧，结合动态焦点状态调整存储优先级；
- �� 由写入VLM生成结构化实体和动作记录，存入长时记忆链；
- �� 结构化记忆包括实体身份、位置、关系、状态等信息；
- �� 视觉证据缓存存储关键帧，支持后续恢复；
- �� 查询时，结构化匹配结合语义扩展，快速定位目标；
- �� 多轮交互中，前轮查询更新焦点状态，持续追踪目标；
- �� 规则驱动写入策略，平衡信息丰富性和存储成本。
Experiments
采用OVO-Bench和StreamingBench两个公开数据集，评估感知和回溯准确率。对比多种SOTA模型，验证FOLIO在多轮长时依赖和目标追踪中的优越性。设置不同存储参数和焦点策略，进行消融分析。指标包括准确率、存储成本和响应时间，确保系统在高性能和实时性之间取得平衡。
Results
FOLIO在OVO-Bench达到82.0/69.1的感知/回溯准确率，超越现有模型，且在StreamingBench上达74.5%的整体准确率。多轮问答中，结合焦点状态和语义扩展的检索策略，显著改善目标追踪和长时依赖理解。消融实验显示，焦点引导和结构化记忆的结合是性能提升的关键。系统在降低存储成本的同时，实现了长时依赖的高效管理。
Applications
该系统适用于自动驾驶、智能监控、机器人助手等场景，能实时理解复杂场景中的目标和事件。只需有限存储资源，即可实现长时间连续监控和多轮交互，极大提升场景感知能力。未来，结合多模态信息和强化学习，有望实现更智能、更高效的连续场景理解。
Limitations & Outlook
系统在极端动态或复杂场景下，焦点状态可能难以准确更新，导致信息遗漏或误追踪。规则驱动的存储策略缺乏自适应能力，可能在不同场景表现不佳。长时记忆在极端长序列中仍面临信息稀释或错配风险，未来需引入学习机制优化存储策略和多模态融合。
Plain Language Accessible to non-experts
想象你在厨房做饭。每次准备食材时，你只记住重要步骤，比如切菜、炒菜，而背景的细节，比如调料、碗碟，都会尽量简化。等到告诉朋友你做了什么菜时，你会重点描述关键步骤和主要食材，而不是所有细节。这就像FOLIO系统一样，它会优先存储重要的实体和动作，把背景信息压缩，确保在需要时能快速找到关键内容。这样既节省空间，又能迅速回答问题。
ELI14 Explained like you're 14
想象你在看一场足球比赛，你只记住了关键的事情，比如谁进球、谁在跑、裁判的判罚。其他细节，比如场边广告、观众反应，虽然也在场，但你不会一直记得全部。当有人问你比赛情况时，你会重点说出那些重要瞬间和人物，而不是所有琐碎的细节。这就像FOLIO一样，它会专注于重要目标，比如球员和动作，把其他背景信息压缩存储。这样，你可以在比赛结束后，快速讲述精彩瞬间，而不用记住所有细节。
Glossary
Structured Memory (结构化记忆)
一种以实体、动作、关系组织的记忆方式，便于长时追踪和检索。
用于存储视频中目标的状态和关系。
Focus State (焦点状态)
动态调整的优先级指标，用于引导存储重要实体和动作。
决定存储资源的分配。
Visual Evidence Cache (视觉证据缓存)
存储关键帧的缓存，用于后续恢复和验证。
支持检索和问答。
Training-free System (无训练系统)
无需端到端训练，通过规则和特征驱动机制实现目标。
系统设计的核心思想。
Multi-turn Interaction (多轮交互)
多次问答中，前轮信息影响后续存储和检索。
实现长时依赖追踪。
Open Questions Unanswered questions from this research
- 1 如何自适应焦点状态以应对极端动态场景？
- 2 多模态信息融合在长时记忆中的潜力？
- 3 系统在超长序列中的性能极限和优化策略？
Applications
Immediate Applications
智能监控
实时追踪目标，压缩存储关键事件，提升监控效率。
自动驾驶
理解道路场景中的关键目标和动作，支持决策。
Long-term Vision
场景理解平台
结合多模态信息，实现连续、全场景理解，推动自动化发展。
Abstract
In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.