Motion-I2V: Consistent and Controllable Image-to-Video Generation with Explicit Motion Modeling
Motion-I2V combines diffusion-based motion prediction with an enhanced temporal attention mechanism to produce consistent, controllable videos from images, supporting sparse trajectories and zero-shot translation.
Key Findings
Methodology
Motion-I2V decomposes image-to-video generation into two stages: first, a diffusion-based motion field predictor estimates pixel-wise trajectories conditioned on reference image and text prompts; second, a motion-augmented temporal attention module propagates features guided by predicted trajectories, enlarging the temporal receptive field. The framework employs a sparse trajectory ControlNet for user-controlled motion editing and supports region-specific animation and zero-shot video translation. Training utilizes large-scale WebVid-10M data, with pre-trained Stable Diffusion and AnimateDiff models, optimizing for trajectory accuracy and frame consistency. The approach effectively addresses large motions and viewpoint variations, ensuring high temporal coherence and controllability.
Key Results
- On a diverse test set of 80 images across multiple categories, Motion-I2V achieved a prompt-following score of 34.86, surpassing VideoComposer (32.62), and maintained high frame consistency (0.9871). In large motion scenarios, the average displacement reached 20.06 pixels, demonstrating superior motion realism. Ablation studies confirmed that incorporating explicit motion prediction and attention-based feature warping significantly improved stability and visual quality. User controls via sparse trajectories and region masks enabled precise editing, with results showing high fidelity and flexibility.
- Quantitative comparisons with SOTA methods like I2VGen-XL and DynamiCrafter showed that Motion-I2V better follows textual prompts and produces more temporally consistent videos, especially under large motions. The ablation experiments highlighted the importance of the two-stage design and motion augmentation, reducing average motion displacement and enhancing stability. The model's ability to support zero-shot video translation further broadens its practical utility, enabling style transfer and content editing without retraining.
- The experimental results demonstrate that the proposed framework effectively balances motion authenticity, detail preservation, and controllability. It outperforms existing approaches in prompt adherence, motion magnitude, and temporal coherence, making it suitable for diverse applications such as content creation, virtual avatars, and AR/VR experiences. The combination of explicit motion modeling and advanced attention mechanisms marks a significant step forward in open-domain video synthesis.
Significance
This work addresses fundamental challenges in large-motion, viewpoint-invariant video synthesis, bridging the gap between high-quality image generation and dynamic video creation. By explicitly modeling motion trajectories and enhancing temporal receptive fields, Motion-I2V achieves unprecedented levels of consistency and controllability. Its ability to interpret sparse user inputs and support zero-shot translation opens new avenues for interactive content creation, virtual reality, and animation industries. The framework's modular design and reliance on large-scale pre-trained diffusion models also facilitate future extensions, such as multi-modal conditioning and real-time applications. Overall, this research significantly advances the state-of-the-art in open-domain, controllable video synthesis, with broad implications for both academia and industry.
Technical Contribution
The paper introduces a diffusion-based motion field predictor that explicitly estimates pixel trajectories, decoupling motion modeling from frame synthesis. It innovates by integrating a motion-augmented temporal attention mechanism, which warps reference features according to predicted trajectories, effectively enlarging the temporal receptive field and improving temporal coherence. The use of a sparse trajectory ControlNet enables fine-grained user control over motion and regions, while the two-stage framework allows stable, high-quality video generation. This approach surpasses prior methods that rely solely on end-to-end training or naive feature warping, providing a flexible, interpretable, and controllable solution for large-motion scenarios.
Novelty
This work is the first to combine explicit pixel-wise motion prediction with diffusion models for open-domain image-to-video synthesis. Its core innovation lies in the motion-augmented temporal attention, which dynamically warps features based on predicted trajectories, significantly improving temporal consistency under large motions. Additionally, the integration of sparse trajectory ControlNet for user-guided editing and zero-shot translation distinguishes it from prior end-to-end models, offering unprecedented controllability and flexibility. These contributions collectively push the boundaries of current video synthesis techniques, enabling more realistic, controllable, and diverse outputs.
Limitations
- Despite improvements, the model's accuracy in predicting complex or highly non-rigid motions remains limited, especially in scenes with occlusion or fast dynamics.
- Training requires extensive video datasets and computational resources, hindering real-time deployment.
- The reliance on sparse user inputs may limit control precision in highly detailed or cluttered scenes, necessitating further robustness enhancements.
Future Work
Future research will focus on integrating multi-modal cues such as depth and semantics to improve motion prediction accuracy. Efforts will also aim to optimize inference speed for real-time applications and extend the framework to higher resolutions. Exploring self-supervised learning strategies could reduce data dependency, while incorporating more sophisticated user controls, like interactive editing and multi-region guidance, will enhance usability. Additionally, expanding to multi-view and 3D-aware generation could further broaden the framework's applicability in AR/VR and content creation.
AI Executive Summary
Motion-I2V introduces a novel two-stage framework for high-quality, controllable image-to-video synthesis, leveraging explicit motion modeling and diffusion-based generation. The first stage employs a diffusion-based motion field predictor to estimate pixel-wise trajectories conditioned on reference images and text prompts, effectively capturing large motions and viewpoint changes. The second stage enhances temporal coherence by applying a motion-augmented temporal attention mechanism, which warps reference features guided by predicted trajectories, thereby enlarging the temporal receptive field and reducing artifacts. This design addresses the limitations of existing methods that rely solely on 1-D temporal attention, which struggle with large motions and lack fine control.
The framework incorporates a sparse trajectory ControlNet, enabling users to specify precise motion paths or regions for animation, significantly improving controllability. It also supports zero-shot video translation, allowing style transfer and content editing without retraining. Extensive experiments on the WebVid-10M dataset demonstrate that Motion-I2V outperforms state-of-the-art approaches in prompt adherence, temporal consistency, and motion realism. Quantitative metrics show a prompt-following score of 34.86, surpassing previous methods, with an average motion displacement of 20.06 pixels in large motion scenes.
This work advances the field by decoupling motion prediction from frame synthesis, providing a flexible, interpretable, and user-controllable solution for diverse scenarios. Its ability to handle large motions, viewpoint variations, and sparse controls makes it a significant step toward practical, high-fidelity video generation. Future directions include multi-modal conditioning, real-time inference, and higher resolution outputs, promising broad impacts in content creation, virtual reality, and animation industries.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
We introduce Motion-I2V, a novel framework for consistent and controllable image-to-video generation (I2V). In contrast to previous methods that directly learn the complicated image-to-video mapping, Motion-I2V factorizes I2V into two stages with explicit motion modeling. For the first stage, we propose a diffusion-based motion field predictor, which focuses on deducing the trajectories of the reference image's pixels. For the second stage, we propose motion-augmented temporal attention to enhance the limited 1-D temporal attention in video latent diffusion models. This module can effectively propagate reference image's feature to synthesized frames with the guidance of predicted trajectories from the first stage. Compared with existing methods, Motion-I2V can generate more consistent videos even at the presence of large motion and viewpoint variation. By training a sparse trajectory ControlNet for the first stage, Motion-I2V can support users to precisely control motion trajectories and motion regions with sparse trajectory and region annotations. This offers more controllability of the I2V process than solely relying on textual instructions. Additionally, Motion-I2V's second stage naturally supports zero-shot video-to-video translation. Both qualitative and quantitative comparisons demonstrate the advantages of Motion-I2V over prior approaches in consistent and controllable image-to-video generation. Please see our project page at https://xiaoyushi97.github.io/Motion-I2V/.