Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generation
UniCaMo constructs a 3D-grounded noise space for joint control of object and camera motion, improving video coherence and quality.
Key Findings
Methodology
UniCaMo builds a shared 3D-grounded noise space by warping Gaussian noise along sparse 3D point tracks for object trajectories, and sampling view-consistent noise from a virtual sphere to handle camera motion. This combines local track-guided noise warping with global sphere-based sampling, ensuring geometric and temporal consistency under both object and camera movements. The approach modifies only the input noise, requiring no architectural changes or additional modules, and leverages lightweight LoRA fine-tuning of large pretrained models like Wan 2.1 (14B).
Key Results
- On MoveBench, UniCaMo achieves a state-of-the-art FID of 10.36 and EPE of 2.3, outperforming baselines such as Wan-Move and other control methods, especially in complex multi-object and multi-view scenarios.
- The method maintains high visual quality, with improved consistency in object trajectories and camera viewpoints, reducing artifacts and geometric inconsistencies.
- Ablation studies confirm that combining sparse 3D point tracks with sphere-based noise sampling significantly enhances control fidelity and scene coherence, especially during large viewpoint changes.
Significance
This work advances controllable video generation by enabling precise, joint control of object and camera motion within a unified framework. It addresses longstanding challenges of motion ambiguity and geometric inconsistency, providing a scalable, architecture-agnostic solution that can be integrated into existing large-scale diffusion models. The approach opens new avenues for realistic scene synthesis in virtual reality, animation, and robotics, where accurate dynamic control is crucial.
Technical Contribution
UniCaMo introduces a novel motion-consistent noise representation that combines sparse 3D point tracks with a spherical noise sampling scheme, enabling joint control of object and camera motion solely through input noise modification. It leverages geometric reasoning, depth-aware occlusion handling, and view-consistent noise sampling, all without architectural modifications. The method's compatibility with large pretrained models via lightweight LoRA fine-tuning makes it both efficient and versatile, setting a new standard for controllable diffusion-based video synthesis.
Novelty
This is the first approach to embed joint object and camera motion control directly into the input noise space of diffusion models using a 3D-grounded, motion-consistent noise representation. Unlike prior methods relying on 2D guidance or additional modules, UniCaMo employs a unified spherical noise sampling combined with sparse 3D point tracks, effectively resolving occlusion and view-dependent artifacts, and enabling precise multi-object, multi-view control in a single, architecture-agnostic framework.
Limitations
- The method depends on accurate sparse 3D point tracks and depth estimation; inaccuracies can lead to artifacts or inconsistent motion.
- Extreme viewpoint changes or highly dynamic scenes may still produce artifacts or geometric inconsistencies.
- While lightweight, LoRA fine-tuning and noise warping add computational overhead, which could be further optimized for real-time applications.
Future Work
Future directions include improving robustness to noisy or incomplete 3D data, extending the framework to real-time interactive control, and integrating multi-modal cues such as audio or language for richer scene synthesis. Additionally, exploring end-to-end training of the entire pipeline could further enhance control fidelity and scene realism, pushing the boundaries of controllable, high-fidelity video generation.
AI Executive Summary
Controllable video synthesis has become a vital area in computer vision, enabling applications from virtual reality to animation. Despite recent advances in large-scale diffusion models, achieving precise, unified control over both object and camera motion remains challenging. Existing methods often treat these aspects separately, leading to motion ambiguity and geometric inconsistencies, especially during complex scene changes. To address this, UniCaMo introduces a novel approach that constructs a shared 3D-grounded noise space, allowing explicit encoding of object trajectories and camera viewpoints.
The core innovation lies in combining sparse 3D point tracks with a spherical noise sampling scheme. The point tracks warp the reference frame's Gaussian noise along desired object paths, while the spherical sampling ensures globally consistent noise values for newly revealed regions during camera motion. This dual mechanism maintains geometric and temporal coherence, even under large viewpoint changes. Crucially, UniCaMo modifies only the input noise, avoiding architectural changes or additional modules, and leverages lightweight LoRA fine-tuning of large pretrained models like Wan 2.1.
Experimental results on MoveBench demonstrate that UniCaMo outperforms existing methods in both visual quality and control accuracy. It achieves a FID of 10.36, surpassing baselines such as Wan-Move, and exhibits superior scene consistency in complex multi-object and multi-view scenarios. Ablation studies confirm that the combination of sparse 3D point guidance and sphere-based sampling effectively resolves occlusion and disocclusion issues, ensuring high-fidelity scene synthesis.
This work significantly advances the field by providing a scalable, architecture-agnostic framework for joint object and camera control. Its ability to generate realistic, coherent videos with complex motions opens new possibilities for immersive virtual environments, film production, and robotics. Future research will focus on enhancing robustness, reducing computational costs, and enabling real-time interactive control, further broadening the impact of this technology.
Deep Dive
Abstract
Modern image-and-text-to-video diffusion models can synthesize highly realistic videos by iteratively denoising an initial Gaussian noise tensor conditioned on reference image and text inputs. However, existing approaches still lack precise and unified controllability over both object motion and camera motion within a single generation process. We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints by directly constructing the input noise of the diffusion model. Specifically, UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames. Sparse 3D point tracks are used to warp the Gaussian noise of the reference frame along desired object trajectories, while a virtual spherical noise representation provides globally consistent noise values for newly revealed scene regions under camera motion. By combining local track-guided noise warping with global sphere-based noise sampling, UniCaMo maintains geometric and temporal consistency under both object movement and viewpoint changes. Because UniCaMo modifies only the input noise, it requires no auxiliary adapters, control branches, or architectural changes to the underlying video diffusion model. With lightweight LoRA fine-tuning on large pretrained video diffusion models, including Wan 2.1 (14B), UniCaMo achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.