LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation
State-conditioned memory router (LayerRecall) enhances long-range content recovery in video diffusion models, achieving top performance on MemoBench and MovieBench.
Key Findings
Methodology
This paper introduces LayerRecall, which analyzes layer-wise temporal preferences in video DiT models to design a layer-selective memory routing mechanism. It retrieves relevant historical key-value (KV) states conditioned on the current context and injects them only into designated memory-sensitive layers, preserving local attention elsewhere. The training employs Cross-Horizon Prediction Matching (CHPM), which supervises the bounded-memory router using a privileged long-context reference, aligning its predictions with those of a long-context teacher. Extensive evaluations on MemoBench, MovieBench, and VBench-Long demonstrate superior long-range content recovery and stability, outperforming existing methods.
Key Results
- Across 100 multi-shot prompts, LayerRecall achieves the highest overall scores on MemoBench and MovieBench, with improvements of over 5% in memory consistency metrics. It effectively restores subject identities and scene details after long temporal gaps, surpassing baseline models in long-distance reappearance scenarios.
- On VBench-Long, the model maintains comparable performance to the backbone DiT, confirming its ability to enhance long-term memory without sacrificing local motion and appearance quality. The layer-specific injection reduces content drift and scene dislocation.
- Ablation studies show that CHPM training improves memory routing accuracy by 0.03 points over random initialization. The layer preference analysis reveals that selective injection based on backbone profiling yields better overall performance, validating the importance of where to inject memory.
Significance
This work addresses a fundamental challenge in long-horizon video synthesis—maintaining semantic and visual consistency over extended durations. By enabling models to selectively retrieve and inject relevant historical information, it significantly reduces content drift and scene inconsistency. The proposed approach is scalable, transferable across architectures, and compatible with existing diffusion frameworks, paving the way for more coherent, long-duration virtual content creation. Its impact extends to virtual reality, film production, and AI-driven content editing, where long-term coherence is crucial.
Technical Contribution
The paper introduces a novel layer-wise analysis revealing distinct temporal preferences in DiT layers. It proposes a state-conditioned, layer-selective memory routing mechanism that dynamically retrieves relevant historical KV states. The training employs a novel CHPM strategy, aligning the bounded-memory model's predictions with a long-context teacher without requiring explicit memory labels. The approach offers a flexible, efficient, and transferable framework for long-range memory utilization in autoregressive video diffusion, with theoretical and practical advantages over prior global or fixed-anchor methods.
Novelty
This is the first work to systematically analyze layer-wise temporal preferences in video diffusion models and leverage this insight to design a layer-selective memory routing mechanism conditioned on current states. Unlike previous methods that rely on fixed memory anchors or dense global memory, LayerRecall dynamically selects and injects relevant historical information into specific layers, enabling more precise long-distance content preservation. The use of CHPM for training without explicit memory labels further distinguishes this approach.
Limitations
- Despite improvements, the model still exhibits some content drift in extremely long scenes, especially where scene complexity exceeds training data diversity. Its performance depends on the quality and quantity of long-horizon videos available for training.
- Training and inference incur additional computational costs due to the retrieval and routing mechanisms, though these are minimal compared to overall model complexity.
- The current approach relies on backbone-specific profiling for layer selection, which may require adaptation when deploying on new architectures or tasks. Future work should explore more generalizable and scalable memory routing strategies.
Future Work
Future directions include integrating multi-modal cues (e.g., audio, text) to enhance memory relevance, developing adaptive layer profiling that dynamically adjusts to scene complexity, and optimizing inference speed for real-time applications. Extending the framework to other generative tasks, such as 3D scene synthesis or interactive editing, is also promising. Additionally, exploring unsupervised or weakly supervised training paradigms could reduce data dependency and improve scalability.
AI Executive Summary
Long-horizon video generation remains a significant challenge in AI, primarily due to the difficulty of maintaining semantic and visual consistency over extended durations. Traditional autoregressive models often rely on recency-based caching, which supports local continuity but fails when subjects, objects, or scene attributes reappear after long gaps, leading to identity drift and scene inconsistencies. Addressing this, the paper introduces LayerRecall, a novel state-conditioned, layer-selective memory routing mechanism that dynamically retrieves relevant historical information conditioned on the current generation state.
The core innovation lies in analyzing the layer-wise temporal preferences within video Diffusion Transformers (DiTs). This analysis reveals that different layers favor different temporal contexts—some prefer recent information, others distant history. Leveraging this insight, LayerRecall selectively injects historical key-value states into memory-sensitive layers, preserving local attention elsewhere. To train this mechanism without requiring expensive long-horizon videos or explicit memory labels, the authors propose Cross-Horizon Prediction Matching (CHPM). This strategy uses a privileged long-context reference to supervise the bounded-memory router by aligning its predictions with a long-context teacher.
Extensive experiments across multiple benchmarks demonstrate that LayerRecall significantly outperforms existing methods in long-range content recovery, identity preservation, and scene consistency. On MemoBench and MovieBench, it achieves over 5% improvement in key metrics, while maintaining comparable performance on VBench-Long. Qualitative analyses show that the model can self-correct mismatched attributes by recalling historical appearances, without disrupting ongoing motion or scene structure. The approach also exhibits good transferability across different backbone architectures and negligible inference overhead.
Overall, this work advances the state of the art in long-horizon video synthesis, offering a scalable, effective, and generalizable solution for long-term semantic consistency. Its implications extend to virtual reality, film production, and AI-driven content creation, where maintaining coherence over extended durations is essential. Future research will focus on integrating multi-modal cues, optimizing inference efficiency, and expanding to other generative domains, promising a new era of coherent, long-duration virtual content.
Deep Dive
Abstract
Autoregressive video diffusion enables scalable long-video generation by producing chunks from a bounded recent context. While recency-based caching preserves local continuity, it evicts historical cues needed when subjects, objects, scenes, or attributes reappear. Existing memory mechanisms expose models to nonlocal history, but access alone does not ensure effective use. Our analysis reveals that video DiT layers exhibit distinct preferences for current, recent, and distant context, suggesting that long-range memory requires deciding both what to retrieve and where to use it. We introduce LayerRecall, a current-conditioned, layer-selective memory router that retrieves relevant historical K/V states and injects them only into backbone-specific memory-sensitive layers while preserving local attention elsewhere. To reduce reliance on scarce high-quality long-horizon videos and explicit memory-allocation labels, we further propose Cross-Horizon Prediction Matching (CHPM), which uses a privileged long-context reference to supervise the bounded-memory router in prediction space. Across 100 multi-shot evaluation prompts, LayerRecall achieves the best overall results on MemoBench and MovieBench while matching its backbone on VBench-Long, demonstrating stronger long-range recovery without sacrificing local continuity. Qualitative analyses further reveal memory-guided self-correction, whereby initially mismatched local attributes return to their historical appearance without resetting ongoing motion or scene structure. Additional analyses show cross-backbone portability and negligible inference overhead.