XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model
XMem employs a multi-store memory framework inspired by human cognition, achieving state-of-the-art long-video object segmentation with low GPU usage.
Key Findings
Methodology
XMem integrates three independent but interconnected memory modules: sensory memory updated each frame, high-resolution working memory, and a compact long-term memory. Memory potentiation consolidates active working memory elements into long-term memory, preventing memory explosion. The novel space-time memory reading mechanism enables efficient retrieval across long sequences. The architecture mimics human memory systems, allowing high-accuracy segmentation on long videos while maintaining low resource consumption.
Key Results
- On the Longtime Video Dataset, XMem achieves a J&F score exceeding 85%, outperforming previous methods by over 10%. On DAVIS and YouTubeVOS, it matches state-of-the-art performance, confirming robustness across short and long videos.
- GPU memory usage remains below 1.4GB, enabling processing of over 30,000 frames, demonstrating excellent scalability.
- Memory consolidation and activation algorithms effectively reduce performance decay over time, validating the approach's long-term stability.
Significance
This work addresses the critical challenge of long-term video understanding by proposing a scalable, efficient memory management framework. It overcomes the limitations of single-memory models that struggle with memory explosion and accuracy degradation, paving the way for real-world applications like surveillance, autonomous driving, and content editing where long video sequences are common.
Technical Contribution
The paper introduces a multi-tiered memory system with dedicated consolidation and potentiation algorithms, enabling stable long-term information storage. The space-time memory reading mechanism enhances feature matching robustness. These innovations collectively allow high-precision segmentation with minimal resource overhead, representing a significant leap over existing single-memory approaches.
Novelty
This is the first approach to explicitly model and implement a multi-store memory system for video segmentation, inspired by human cognition theories. Unlike prior methods that rely on eager compression or single memory banks, XMem’s architecture effectively balances long-term retention and short-term adaptability, enabling long video processing with high accuracy.
Limitations
- The model’s performance diminishes in scenarios with extreme occlusion or rapid motion due to limitations in memory fusion robustness. Handling such cases requires further algorithmic refinement.
- Processing ultra-long videos still depends on hardware capabilities; although memory is controlled, computational costs can be high for very long sequences.
- Training relies heavily on annotated datasets, and generalization to unseen scenarios remains a challenge. Further research into unsupervised or semi-supervised learning could address this.
Future Work
Future directions include enhancing memory fusion robustness, integrating multi-modal cues like audio and text, and optimizing hardware implementation for edge devices. Exploring self-supervised learning to reduce annotation dependence and improve generalization also holds promise.
AI Executive Summary
Video object segmentation is a fundamental yet challenging task in computer vision, especially when dealing with long sequences. Existing methods often rely on single memory modules, which tend to suffer from memory explosion and performance decay over time. These limitations hinder practical deployment in applications such as surveillance, autonomous navigation, and video editing, where processing extended sequences efficiently and accurately is essential.
Inspired by human cognition, this study introduces XMem, a novel architecture that employs a multi-store memory system comprising sensory, working, and long-term memories. The sensory memory captures short-term, frame-specific features and is updated every frame. The working memory stores a subset of historical features, updated periodically, while the long-term memory consolidates the most relevant information through a memory potentiation algorithm. This process ensures a compact, representative set of features that can be efficiently retrieved during segmentation.
The core technical innovation lies in the space-time memory reading mechanism, which combines features from all three memory modules to produce accurate segmentation masks. The memory consolidation process involves selecting prototypes from working memory based on usage frequency, aggregating features via a non-local potentiation method, and managing memory size through least-frequently-used eviction. Experimental results on datasets like Longtime Video, DAVIS, and YouTubeVOS demonstrate that XMem surpasses state-of-the-art methods by over 10% in long video scenarios, while maintaining competitive performance on short videos. GPU memory consumption remains below 1.4GB, enabling scalable long-video processing.
This work significantly advances the field by providing a scalable, resource-efficient framework for long-term video understanding. Its human-inspired multi-memory design offers a promising pathway for future research in video analysis, multi-modal integration, and real-time applications. Despite some limitations in handling extreme occlusions and rapid motions, the approach opens new avenues for robust, long-duration video segmentation, with broad implications for both academia and industry.
Deep Analysis
Background
Video目标分割是计算机视觉中的核心任务之一，早期方法依赖光流和模板匹配，逐步发展到深度学习模型如OSVOS、STM等。STM引入空间-时间记忆机制显著提升性能，但在长视频中存在记忆爆炸和性能衰退的问题。近年来，压缩特征和多尺度策略被提出，但仍难以兼顾长时记忆和高精度。人类记忆模型如Atkinson-Shiffrin提出多存储系统，为解决长视频信息管理提供启示。现有方法在短视频表现良好，但在长时域中表现有限，成为研究难点。
Core Problem
长视频目标分割面临存储资源有限和信息持续性不足的双重挑战。单一记忆模型在长时间跨度内易出现记忆爆炸或信息遗失，导致性能下降。如何在保证高精度的同时，有效管理海量信息，成为核心难题。此外，现有方法在长视频处理时资源消耗大，难以满足实际应用需求。
Innovation
提出多存储记忆架构，模仿人类多系统记忆机制，结合感官、工作和长远记忆。引入记忆激活算法，定期将工作记忆中的重要信息融合到长远记忆，避免爆炸。设计空间-时间记忆读取机制，提高信息匹配鲁棒性。模型在保持低GPU占用的同时，显著提升长视频目标分割性能，突破传统单一记忆模型的局限。
Methodology
- �� 初始化：用第一帧和目标掩码建立三类记忆。• 感官记忆：每帧更新，捕获短期信息。• 工作记忆：每r帧插入新特征，存储历史信息。• 长远记忆：定期融合工作记忆中的代表性元素，利用记忆激活算法生成原型。• 记忆读取：结合空间-时间机制，从三类存储中提取特征。• 目标分割：利用读取的特征和解码器生成掩码。• 记忆管理：超出容量时，淘汰低使用频率元素，确保资源有限。• 训练：在合成和真实数据上预训练，采用交叉熵和Dice损失优化。
Experiments
在YouTubeVOS、DAVIS和Longtime Video Dataset上进行评估，采用J&F指标。模型超参数包括Tmax=10，P=128，k=30。对比SOTA方法如STM、AOT，验证长短视频性能。通过消融实验分析记忆融合、激活机制的贡献。模型在长视频中保持优异表现，GPU内存控制在1.4GB以内，验证其实用性。
Results
XMem在长视频数据集J&F指标达85%以上，比STM等方法提升10%以上。在短视频上性能与最优模型相当，验证其多场景适应性。记忆管理机制有效缓解了长视频中的性能衰退，模型能处理超过3万帧，显示出极强的扩展性。实验证明多存储架构在长时目标追踪中的优势，推动长视频理解技术发展。
Applications
可应用于视频监控、自动驾驶、内容编辑等场景，尤其适合长时间视频分析。模型能在有限硬件资源下实现高效目标追踪，提升自动化水平。未来结合多模态信息，将拓展到多任务视频理解和智能监控系统。
Limitations & Outlook
模型在极端遮挡和快速运动场景下仍存在性能下降，原因在于记忆融合策略的鲁棒性不足。长视频处理受硬件限制，超长视频可能出现性能瓶颈。训练依赖大量标注数据，泛化能力仍需验证。未来将优化记忆融合算法，结合自监督学习，增强模型适应性。
Plain Language Accessible to non-experts
想象你在厨房做饭，厨房里有不同的储藏柜：一个快速拿取的调料柜（感官记忆），一个存放常用食材的柜子（工作记忆），以及一个存放长时间储备的仓库（长远记忆）。每次做菜时，你会先从调料柜拿出调料，偶尔会把常用食材放到工作柜，长时间的储备则在仓库里。做饭过程中，你不断调整调料和食材的使用，偶尔会把重要的调料存到仓库里备用。这个过程类似于XMem模型中的多层次记忆系统，帮助它在处理长视频时，既能快速反应，又能记住长远信息，保证目标分割的准确性。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，比如需要记住很多任务和路线。你会用不同的笔记本：一个快速记下刚看到的线索（感官记忆），一个整理最近任务的笔记本（工作记忆），还有一个存放重要线索和长远计划的宝箱（长远记忆）。每当你完成一段任务，就会把重要的线索从笔记本转存到宝箱里，确保以后还能用。游戏中遇到新线索时，你会快速记下来，偶尔整理一下，把重要信息存到宝箱。这个方法让你在长时间游戏中都能记住关键信息，不会忘记重要线索，也不会被信息淹没。
Glossary
Memory Potentiation (记忆激活)
一种将工作记忆中的重要信息整合到长远记忆中的算法，确保信息不会遗失或过时。
用于模型中定期将活跃的工作记忆元素融合入长远记忆。
Space-Time Memory Reading (时空记忆读取)
一种结合空间和时间信息，从多存储中提取相关特征的机制，用于目标匹配。
实现对长视频中目标的高效追踪。
Memory Consolidation (记忆巩固)
将多次使用的工作记忆片段压缩整合成代表性原型，存入长远记忆。
避免记忆爆炸，保持模型长时间运行能力。
Prototypes (原型)
从多个记忆元素中抽取的代表性特征，用于长远记忆存储。
通过原型机制实现高效记忆压缩。
Attention Mechanism (注意力机制)
一种根据相关性动态调整信息权重的机制，用于提升特征匹配效果。
在记忆读取中实现目标的准确匹配。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端遮挡和快速运动场景下的鲁棒性仍待探索。
- 2 长视频中多模态信息融合（如声音、文本）对目标追踪的影响未充分研究。
Abstract
We present XMem, a video object segmentation architecture for long videos with unified feature memory stores inspired by the Atkinson-Shiffrin memory model. Prior work on video object segmentation typically only uses one type of feature memory. For videos longer than a minute, a single feature memory model tightly links memory consumption and accuracy. In contrast, following the Atkinson-Shiffrin model, we develop an architecture that incorporates multiple independent yet deeply-connected feature memory stores: a rapidly updated sensory memory, a high-resolution working memory, and a compact thus sustained long-term memory. Crucially, we develop a memory potentiation algorithm that routinely consolidates actively used working memory elements into the long-term memory, which avoids memory explosion and minimizes performance decay for long-term prediction. Combined with a new memory reading mechanism, XMem greatly exceeds state-of-the-art performance on long-video datasets while being on par with state-of-the-art methods (that do not work on long videos) on short-video datasets. Code is available at https://hkchengrex.github.io/XMem