StoryMem: Multi-shot Long Video Storytelling with Memory
StoryMem employs Memory-to-Video (M2V) to convert pretrained single-shot diffusion models into multi-shot storytelling tools, significantly improving cross-shot consistency with 28.7% gain on ST-Bench.
Key Findings
Methodology
This paper introduces the Memory-to-Video (M2V) framework, integrating explicit visual memory with single-shot diffusion models through a dynamically updated keyframe bank. Semantic keyframes are selected via CLIP features, filtered by aesthetic scores (HPSv3), and injected into the model using latent concatenation and negative RoPE shifts. Fine-tuning involves only LoRA parameters, enabling efficient adaptation. The approach maintains high visual quality while ensuring long-range coherence across multiple shots, supporting scene transitions and personalized controls.
Key Results
- On the ST-Bench benchmark, StoryMem surpasses prior methods like HoloCine and independent shot models, achieving a cross-shot consistency score of 0.5065, a 28.7% improvement over the baseline. The generated videos exhibit consistent characters, seamless scene transitions, and adherence to story scripts, demonstrating superior storytelling coherence.
- Ablation studies confirm that semantic keyframe selection and aesthetic filtering are crucial for stable memory content. The model's ability to generate minute-long, multi-shot videos with high aesthetic fidelity and script accuracy marks a significant advancement in long-form video synthesis.
- Fine-tuning only 0.7B parameters via LoRA makes the method computationally efficient, adaptable to various scenarios such as scene switching and reference-based generation, broadening practical applications.
Significance
This work addresses the longstanding challenge of maintaining visual and narrative coherence in long video generation. By mimicking human memory, it enables models to remember salient details over extended durations, paving the way for automated storytelling, virtual production, and personalized content creation. The approach balances efficiency and quality, making long-form video synthesis feasible at scale, and opens new avenues for multimodal content generation with high fidelity and coherence.
Technical Contribution
The paper introduces a novel memory-conditioned diffusion framework, combining latent memory injection with negative RoPE shifts to handle discrete keyframes. It leverages lightweight LoRA fine-tuning on pretrained models, avoiding costly retraining. The semantic filtering strategy ensures informative memory, while the hybrid memory management (sink + sliding window) maintains stability. This architecture supports scene transitions and reference-guided generation, representing a significant technical leap in scalable, coherent long video synthesis.
Novelty
This is the first work to explicitly incorporate a dynamic visual memory mechanism into general video diffusion models for multi-shot long video storytelling. Unlike prior methods relying solely on full attention or independent keyframes, the proposed M2V framework effectively captures long-term dependencies with minimal computational overhead, enabling high-quality, coherent, minute-long videos. The integration of negative RoPE shifts and semantic filtering further distinguishes this approach.
Limitations
- Despite improvements, the model still faces challenges in extremely complex scenes with rapid scene changes or high motion, where memory filtering may omit critical details. The reliance on pretrained models limits adaptation to highly diverse or novel domains.
- Computational demands remain significant, especially at higher resolutions or longer videos, necessitating further optimization. The current memory update strategy may need refinement for real-time applications.
- Long-term scene consistency can degrade over very extended durations, requiring more sophisticated memory management or reinforcement learning-based updates in future work.
Future Work
Future research will focus on enhancing memory update strategies, integrating multimodal cues like audio and motion, and exploring reinforcement learning to optimize story coherence. Extending the framework to real-time applications, multi-agent scenarios, and interactive storytelling are promising directions. Additionally, developing more efficient architectures to handle higher resolutions and longer durations will be crucial for industrial deployment.
AI Executive Summary
Long-form video storytelling remains a complex challenge in multimedia synthesis, demanding both high visual fidelity and narrative coherence across extended durations. Traditional approaches either rely on full-sequence modeling, which incurs prohibitive computational costs, or on independent keyframe methods that often produce disjointed scenes lacking smooth transitions. Addressing this gap, the authors introduce StoryMem, a novel framework that leverages a Memory-to-Video (M2V) mechanism inspired by human memory. This approach maintains a compact, dynamically updated memory bank of keyframes, which encodes salient characters, scenes, and stylistic cues from previous shots.
The core technical innovation involves integrating semantic keyframe selection via CLIP features, aesthetic filtering through HPSv3, and latent memory injection using latent concatenation combined with negative RoPE shifts. This allows the pretrained single-shot diffusion models to generate coherent multi-shot videos with minimal fine-tuning—only 0.7 billion parameters via LoRA. The system iteratively updates the memory bank after each shot, ensuring long-range consistency and smooth scene transitions.
Experimental results on the newly proposed ST-Bench benchmark demonstrate that StoryMem outperforms existing methods like HoloCine and independent shot models, achieving a 28.7% improvement in cross-shot consistency. The generated videos exhibit high aesthetic quality, character and scene consistency, and adherence to story prompts. This work significantly advances the field of long video synthesis, offering a scalable, efficient, and versatile solution for automated storytelling, virtual production, and personalized content creation.
Looking ahead, the authors plan to incorporate multimodal cues, reinforcement learning, and real-time capabilities to further enhance story coherence and applicability. The framework’s flexibility supports scene transitions, reference-guided generation, and scene-specific customization, promising broad impact across entertainment, education, and virtual reality industries.
Deep Analysis
Background
近年来，视频生成技术取得了快速发展，尤其是在短视频和单镜头场景中，基于扩散模型的生成方法如Stable Diffusion、Imagen Video等已达到较高水平。然而，长视频和多镜头叙事仍面临巨大挑战。全局建模方法如LCT采用全注意力机制，能捕获长距离依赖，但计算成本极高，不适合长时间生成。关键帧分离策略虽然效率较高，但场景切换不自然，人物不一致，限制了实际应用。随着虚拟现实、影视制作等行业对长时间、多场景内容的需求增长，亟需一种兼顾效率和连贯性的新型方法。
Core Problem
核心问题在于如何在保证生成质量的同时，实现长时间、多场景的视觉和叙事连贯。全局建模虽然能捕获长距离依赖，但计算成本高昂，难以扩展到分钟级视频；关键帧方法虽高效，却导致场景切换不自然，人物不一致，影响故事流畅性。现有技术难以在保持高视觉品质的基础上，确保故事的连续性和人物的稳定性，限制了其在实际应用中的推广。
Innovation
本文提出Memory-to-Video（M2V）机制，创新点包括：1）引入显式视觉记忆库，动态筛选语义关键帧，确保记忆内容丰富且稳定；2）结合Latent拼接和负RoPE偏移技术，将记忆信息融入预训练模型，增强长时依赖捕获能力；3）只需微调少量参数（LoRA），避免大规模训练，极大降低成本。该方案兼顾效率与效果，支持多场景扩展，如场景切换和参考图像引导，突破了以往单镜头模型的局限。
Methodology
- �� 构建基于预训练单镜头扩散模型（Wan2.2-I2V），利用Mask引导生成。
- �� 设计Memory-to-Video（M2V）机制，将关键帧存入记忆库，动态筛选语义关键帧（CLIP特征）和美学过滤（HPSv3）。
- �� 采用Latent拼接，将记忆Latent与视频Latent融合，结合负RoPE偏移，处理记忆与当前镜头的时间关系。
- �� 通过微调LoRA参数，优化模型适应多镜头长视频生成。
- �� 在每次生成后，更新记忆库，确保记忆内容的代表性和稳定性。
- �� 支持场景切换和参考图像引导，增强多样性和个性化。
Experiments
在ST-Bench上，使用多场景、多镜头脚本，评估跨镜头一致性、视觉质量和脚本符合度。对比基线包括单镜头模型、关键帧扩展和HoloCine。参数设置包括：记忆长度1-10帧，负RoPE偏移S=5，微调参数约0.7B。通过用户主观评价和ViCLIP指标，验证方法优越性。还进行了消融实验，验证记忆筛选和过滤策略的效果。
Results
实验显示，StoryMem在跨镜头一致性指标上达0.5065，优于HoloCine（0.4628）和单镜头模型（0.2452），提升约28.7%。视觉质量和故事符合度也显著优于对比方法。记忆筛选和过滤策略有效减少信息冗余，确保记忆内容丰富且稳定。微调参数少，训练成本低，适应性强，支持多场景扩展。
Applications
该技术可应用于自动影视制作、虚拟主播、虚拟现实内容创作等场景，尤其适合长时间、多场景的故事叙述。只需提供故事脚本和参考图像，即可生成连贯的多镜头视频，极大降低制作成本。未来还可结合语音、动作等多模态信息，丰富虚拟场景和角色表现。
Limitations & Outlook
当前模型对极端复杂场景和动作的表现仍有限，记忆筛选策略在多样化场景中有待优化。长时间生成时，场景漂移和人物不一致问题仍存在。计算资源需求较高，特别是在高分辨率和多场景切换时，效率仍需提升。未来需加强记忆更新机制和多模态融合能力。
Plain Language Accessible to non-experts
想象你在做一部长篇电影，每个场景都要有连贯的故事和人物。传统的方法就像用一台机器一口气拍完所有场景，既费时又容易出错。另一种办法是只拍几个关键场景，然后拼接起来，但这样场景之间会显得突兀，没有连贯感。本文的方法像是你有一个记忆宝盒，里面存着重要的场景和人物信息，每拍完一个场景，就把它放进去。下一次拍摄时，系统会参考这个宝盒，确保人物不变、场景连贯。通过智能筛选和过滤，宝盒里的内容既丰富又稳定，帮助你拍出一部连贯自然的长片。这就像你有个聪明的助手，总记着所有重要细节，让故事流畅又精彩。这种方法让长时间的故事变得更容易拍，也更好看。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的积木游戏，你要建一座很长的城堡，里面有很多房子、人物和道路。每次你搭完一部分，就要记住它的样子，这样下一次搭建时，城堡才不会变形或出现奇怪的地方。以前的方法就像每次都从头开始拼，或者只记住几个关键房子，结果城堡看起来不连贯。现在，这个新方法像是你有一个神奇的记忆盒子，里面装满了你之前搭好的房子和道路。每次搭建新部分时，它会帮你记住重要的细节，确保每个房子都长得一样，城堡看起来完整又漂亮。只要你告诉它故事的内容，它就能帮你拼出一座长长的、连贯的城堡，像电影一样精彩。这让你不用每次都从零开始，也不用担心城堡会变得乱七八糟。
Abstract
Visual storytelling requires generating multi-shot videos with cinematic quality and long-range consistency. Inspired by human memory, we propose StoryMem, a paradigm that reformulates long-form video storytelling as iterative shot synthesis conditioned on explicit visual memory, transforming pre-trained single-shot video diffusion models into multi-shot storytellers. This is achieved by a novel Memory-to-Video (M2V) design, which maintains a compact and dynamically updated memory bank of keyframes from historical generated shots. The stored memory is then injected into single-shot video diffusion models via latent concatenation and negative RoPE shifts with only LoRA fine-tuning. A semantic keyframe selection strategy, together with aesthetic preference filtering, further ensures informative and stable memory throughout generation. Moreover, the proposed framework naturally accommodates smooth shot transitions and customized story generation applications. To facilitate evaluation, we introduce ST-Bench, a diverse benchmark for multi-shot video storytelling. Extensive experiments demonstrate that StoryMem achieves superior cross-shot consistency over previous methods while preserving high aesthetic quality and prompt adherence, marking a significant step toward coherent minute-long video storytelling.