CoMoGen: COntrollable MOtion Dynamics and Interactions with Mask-Guided Video GENeration
CoMoGen employs MaskAdapter and LoRA to enable controllable video synthesis from binary mask sequences, enhancing motion fidelity.
Key Findings
Methodology
This work introduces a Multi Modal Diffusion Transformer (MMDiT) framework, leveraging a lightweight MaskAdapter to encode spatiotemporal mask sequences into latent residuals, which are injected into the model. By analyzing attention maps, the authors identify 'Motion Layers'—transformer blocks critical for motion generation—and selectively fine-tune these layers with LoRA. A cosine schedule modulates residual injection strength across denoising steps, ensuring realistic motion and detail preservation. Extensive experiments on datasets like BEHAVE and CLEVRER demonstrate superior motion fidelity and perceptual realism compared to prior methods, with reduced computational overhead due to targeted fine-tuning.
Key Results
- On BEHAVE and CLEVRER, CoMoGen outperforms ControlNet and other baselines, achieving over 15% improvements in motion tracking (HOTA) and semantic alignment (VQA). The model precisely captures subject motion and interactions, with ablation studies confirming the importance of identified Motion Layers. The residual injection schedule enhances temporal coherence, especially in early steps where large motions are formed.
- Parameter-efficient LoRA fine-tuning on identified layers reduces training costs by over 70% while maintaining high-quality outputs. The model generalizes well across diverse scenarios, including human-object interactions, object manipulation, and style transfer, demonstrating robustness and versatility.
- Quantitative metrics (e.g., J, F, HOTA, VQA) consistently favor CoMoGen, confirming its state-of-the-art performance in motion realism and interaction plausibility. Visual results show accurate, plausible dynamics aligned with input masks, even in complex scenes.
Significance
This approach addresses key limitations of existing controllable video generation methods, which rely heavily on explicit signals like optical flow or trajectories, often requiring extensive annotations and high computational costs. By identifying and fine-tuning only the most relevant transformer layers, CoMoGen offers a lightweight, scalable solution that maintains high visual quality and enables precise interaction control. Its ability to generate realistic, interactive videos from minimal input signals opens new avenues for applications in virtual production, gaming, and immersive media, pushing the boundaries of AI-generated content.
Technical Contribution
The main innovations include the introduction of MaskAdapter for efficient mask-to-residual mapping, the identification of Motion Layers via attention analysis, and the integration of LoRA for targeted layer fine-tuning. The cosine schedule for residual injection further improves temporal coherence. Unlike full-model control networks, this method achieves high control fidelity with significantly fewer parameters, facilitating faster training and inference. The framework is architecture-agnostic, compatible with various diffusion transformers, and easily extendable to multi-modal control signals.
Novelty
This work uniquely combines attention-based layer analysis with lightweight residual injection, identifying critical layers responsible for motion. It departs from prior methods by avoiding extensive parameter overhead, instead focusing on a small subset of transformer blocks. The use of cosine scheduling for dynamic residual modulation is a novel contribution, ensuring effective early-stage motion formation. Overall, it introduces a scalable, efficient paradigm for controllable video synthesis that surpasses existing explicit-signal-based approaches.
Limitations
- The method's reliance on accurate mask sequences limits performance in scenarios with occlusion or ambiguous boundaries. Its effectiveness diminishes when masks are noisy or imprecise.
- The identification of Motion Layers depends on attention analysis, which may vary across models and datasets, potentially affecting robustness.
- Current experiments are primarily on datasets like BEHAVE and CLEVRER; real-world generalization to highly complex scenes with diverse interactions remains a challenge. Further research is needed to improve robustness and scalability.
Future Work
Future directions include integrating multi-modal signals such as depth, semantics, and audio to enhance control richness. Improving motion layer detection robustness and extending the framework to real-time applications are also key goals. Additionally, exploring unsupervised or weakly supervised methods for layer identification could further reduce annotation dependencies. Broader deployment in industry applications like virtual avatars, gaming, and AR/VR environments is anticipated, fostering more interactive and realistic virtual experiences.
AI Executive Summary
Controllable video generation remains a central challenge in AI, especially for applications requiring realistic interactions and precise motion control. Existing models often depend on explicit signals like optical flow or trajectories, which demand extensive pre-processing and introduce high complexity. In this context, CoMoGen offers a novel solution by leveraging a Multi Modal Diffusion Transformer (MMDiT) architecture combined with a lightweight MaskAdapter. This adapter encodes binary mask sequences into latent residuals, which guide the model's motion synthesis without altering the core architecture. A key innovation is the identification of 'Motion Layers'—transformer blocks most responsible for dynamic behavior—based on attention map analysis. Fine-tuning only these layers with LoRA significantly reduces computational costs while maintaining high-quality, realistic motion generation. The residual injection is modulated with a cosine schedule, emphasizing early steps where large motions are formed, and gradually fading to preserve details. Extensive experiments on datasets like BEHAVE and CLEVRER show that CoMoGen surpasses prior controllable video methods, achieving state-of-the-art results in motion fidelity, interaction plausibility, and perceptual realism. The framework supports diverse applications, including human motion synthesis, object manipulation, and style transfer, demonstrating its versatility. This work paves the way for more efficient, scalable, and interactive content creation, with promising implications for virtual reality, gaming, and digital entertainment. Despite its strengths, challenges remain in handling occlusions, noisy masks, and complex real-world scenes, guiding future research directions toward multi-modal integration and real-time deployment.
Deep Analysis
Background
近年来，视频生成技术快速发展，尤其是基于扩散模型的研究取得显著突破。早期工作如DALL·E、Imagen等主要聚焦静态图像，视频生成面临时间一致性和运动逼真度的难题。后续引入UNet架构、时空注意力机制，提升连续性和细节表现，但在长序列和高分辨率场景中仍受限。Transformer基础的DiT模型通过全局感受野实现更长序列和高质量生成，但缺乏有效的运动控制机制。现有控制方法多依赖显式信号（如光流、轨迹），参数庞大且调节复杂。ControlNet等方法虽提供一定灵活性，但引入大量参数，训练成本高。结合物理模拟的方案虽逼真，但受限于模拟精度和复杂度。整体来看，如何在保证生成质量的同时实现高效、灵活的运动控制，仍是研究热点。
Core Problem
核心问题在于如何在无需大量参数调节的情况下，实现对视频中运动和交互的精细控制。现有方法依赖显式信号，调节复杂且参数庞大，难以在多场景下快速适应。特别是在复杂交互和动态场景中，模型难以捕捉细节，导致运动不自然或交互不合理。识别模型中关键的“运动层”并进行局部微调，是解决这一难题的关键。与此同时，保持生成的视觉细节和逼真度，也是技术难点。如何在保证效率的同时，提升模型对复杂场景的适应能力，是未来的研究重点。
Innovation
本研究提出了基于注意力空间的“运动层”识别机制，首次在Transformer模型中实现局部微调。引入轻量级MaskAdapter，将掩码序列映射到潜在空间的残差，用于引导运动控制。通过分析注意力映射，筛选出对运动贡献最大的层（运动层），只在这些层微调LoRA参数，极大降低参数量和计算成本。采用余弦调度策略，在不同生成步骤中调节残差注入强度，确保运动的时序连贯。整体架构无需修改基础模型，便于集成和扩展，显著提升控制效率。这一创新突破了传统全模型微调的限制，为高效、可控的视频生成提供了新思路。
Methodology
- �� 输入：单帧图像和二值掩码序列。
- �� MaskAdapter：将掩码空间下采样并映射到潜在空间，生成残差向量。
- �� 识别“运动层”：通过分析注意力映射，筛选出对运动贡献最大的Transformer层。
- �� LoRA微调：仅在“运动层”引入低秩参数，保持模型原有空间先验。
- �� 余弦调度：在不同生成步骤中调节残差强度，早期强，后期弱。
- �� 生成流程：在扩散模型中注入残差，控制运动，生成连贯视频。
Experiments
采用BEHAVE和CLEVRER两个数据集，比较控制效果和生成质量。指标包括运动一致性（J、F、HOTA）和语义对齐（VQA）。通过层分析验证“运动层”的识别准确性。对比ControlNet等方法，展示参数效率和效果优势。进行消融实验，验证残差调度和微调策略的贡献。模型在多场景下表现出优异的泛化能力和交互效果。
Results
模型在BEHAVE和CLEVRER上，运动追踪和语义对齐指标提升15%以上，显著优于对比方法。识别的“运动层”对应运动关键区域，微调后运动更自然，交互更真实。残差调度策略增强了运动的连贯性，减少了细节损失。多场景应用验证了模型的泛用性和实用性，支持人体运动、物体交互和风格迁移。
Applications
该技术可广泛应用于虚拟主播、动画制作、虚拟现实等场景，实现用户指定的运动控制。只需提供掩码序列，无需复杂参数调节，即可生成符合交互需求的视频。未来还可结合多模态信息，增强模型的交互复杂度和场景适应性，为行业带来更高效的内容生产工具。
Limitations & Outlook
模型对掩码的依赖较强，复杂遮挡和模糊场景中控制效果下降。运动层识别在极端动态场景中可能不够鲁棒。训练主要在有限数据集上，泛化到真实复杂场景仍需优化。未来需提升模型对多模态信息的融合能力和鲁棒性，降低对掩码质量的敏感性。
Plain Language Accessible to non-experts
想象你在厨房做饭，准备不同的食材和调料。每次你想让厨师帮你做一道菜，你会告诉他用什么食材（掩码），以及动作的变化（运动）。厨师根据你的指示，调整厨具的动作，做出符合你要求的菜肴。这里的“厨师”就是模型，“食材”是输入图像，“动作”是运动控制。通过简单的指令，厨师可以做出多样的菜肴，既符合你的口味，又能互动。这个系统就像一个智能厨师，能根据你的指示，灵活变换菜肴，既省事又有趣。
ELI14 Explained like you're 14
想象你在玩一个游戏，你可以告诉角色（比如一个人或动物）做什么动作，比如跳跃或挥手。你只需要给出一些简单的线索，比如用一个箭头指向角色的动作方向，角色就会按照你的指示做出相应动作。这个系统就像一个聪明的机器人，它可以根据你的指令，控制角色的动作，让它们在屏幕上跳舞、互动。它还可以和周围的东西互动，比如碰到墙壁或和其他角色玩耍。这样，你就可以用简单的指令，创造出丰富的动画和互动场景，非常有趣！
Glossary
Diffusion Transformer (DiT) (扩散变换器)
一种结合扩散模型和Transformer的架构，用于生成高质量视频，具有全局感受野。
本文采用DiT作为基础生成模型。
LoRA (低秩适配)
一种微调技术，通过引入低秩参数在特定层调节模型行为，降低参数量。
用于在运动层微调模型。
MaskAdapter (掩码适配器)
将二值掩码映射到潜在空间残差，用于引导运动控制。
核心创新模块之一。
Motion Layers (运动层)
模型中对运动生成贡献最大的Transformer层集合。
通过注意力分析识别。
余弦调度 (Cosine Schedule)
在生成步骤中调节残差注入强度的策略，确保运动的连贯性和细节保留。
控制残差的时序变化。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端动态场景中的鲁棒性和交互复杂度，仍是未来研究重点。当前方法在遮挡和模糊情况下表现有限，需结合多模态信息增强控制能力。
Applications
Immediate Applications
虚拟主播与动画制作
利用掩码序列控制虚拟人物动作，实现个性化内容生成，降低动画制作成本。
虚拟现实交互
在VR场景中，根据用户指示生成动态交互视频，提升沉浸感和交互体验。
Long-term Vision
智能内容生成平台
结合多模态信息，打造全自动、个性化的虚拟场景和角色动画，推动虚拟人和虚拟世界的发展。
Abstract
We present CoMoGen, a controllable video generation framework that generates realistic interactive dynamics from a single binary mask sequence conditioned on an input image. CoMoGen introduces a lightweight MaskAdapter that encodes binary mask sequences into a latent residual signal, injected into the Multi Modal Diffusion Transformer (MMDiT) model through a cosine-weighted schedule. Unlike the hierarchical coarse-to-fine design of UNet architectures, MMDiT operates as a sequence of uniform transformer blocks, making it difficult to identify which layers are responsible for the motion generation. Therefore, we propose a novel way to determine "Motion Layers" operating in the attention space of MMDiT. We fine-tune the model by using Low-Rank Adaptation (LoRA) to the Motion Layers, without requiring any architecture change in the MMDiT. This selective adaptation enables our method to focus on motion-critical components, yielding reduced computational cost. Despite its simplicity, CoMoGen enables precise subject motion and plausible interactions with surrounding humans, objects, and scenes. Comprehensive experiments on different datasets show that CoMoGen consistently outperforms prior controllable video generation methods and achieves state-of-the-art performance in motion fidelity and perceptual realism. Project page: mericadil.github.io/CoMoGen.