MoCam: Unified Novel View Synthesis via Structured Denoising Dynamics
MoCam employs structured denoising dynamics in diffusion models to unify geometry and appearance for robust novel view synthesis.
Key Findings
Methodology
MoCam introduces a stage-wise conditioning switch within a latent diffusion framework, where early denoising relies solely on geometric scaffolds to establish coarse structure, and later stages switch to source appearance cues for detail refinement. This approach leverages a pre-trained latent space diffusion model, utilizing inverse projection to generate dynamic point clouds as geometric priors. During denoising, the model conditions on these priors initially, then transitions to appearance signals, effectively decoupling geometric errors from appearance refinement. The process avoids error propagation common in static fusion methods, enabling robust handling of sparse or distorted point clouds.
Key Results
- On the OpenVid-1M dataset, MoCam achieves a FVD score of 255.16, outperforming GEN3C (289.37) and TrajectoryCrafter (313.65), demonstrating superior robustness in scenes with severe geometric holes and distortions.
- In single-view 3D reconstruction, MoCam surpasses baseline methods with lower pose errors and higher perceptual quality metrics, accurately reconstructing complex geometries from monocular input.
- For dynamic video re-camera tasks, MoCam maintains high background and subject consistency, with perceptual scores exceeding 0.93, significantly better than ReCam and static approaches, even under large camera motions.
Significance
This work addresses the fundamental challenge of integrating unreliable geometric priors with high-fidelity appearance, especially under large view changes. By decoupling geometry and appearance in a stage-wise manner within diffusion models, MoCam enhances robustness and realism, pushing the boundaries of controllable view synthesis. Its ability to handle sparse, incomplete, or distorted point clouds broadens the applicability in real-world scenarios such as virtual production, immersive media, and AR/VR content creation, offering a scalable solution to longstanding geometric-visual conflicts.
Technical Contribution
The core technical innovation is the stage-wise conditioning switch mechanism within a latent diffusion framework, which dynamically aligns geometric and appearance cues at different denoising stages. This decoupling strategy prevents error accumulation and signal conflicts, enabling stable, high-quality synthesis from imperfect geometry. The method integrates inverse projection-based point cloud generation with a pre-trained latent diffusion model, creating a unified multi-modal conditioning pipeline that adapts to scene sparsity and distortions, representing a significant advancement over static conditioning approaches.
Novelty
This is the first approach to implement a staged conditioning strategy in diffusion models for view synthesis, explicitly decoupling geometric and appearance signals over the denoising timeline. Unlike prior static fusion methods, MoCam’s dynamic switch mechanism effectively mitigates error propagation, providing a new paradigm for robust, high-fidelity novel view generation under challenging geometric conditions.
Limitations
- The method heavily depends on accurate depth estimation; errors here can still affect the final output quality, especially in highly occluded or complex scenes.
- Large viewpoint changes or dynamic scenes with rapid motion may introduce residual artifacts or distortions.
- Training requires extensive paired data and computational resources, limiting immediate scalability for real-time applications.
Future Work
Future directions include integrating self-supervised depth refinement to reduce dependency on initial depth accuracy, exploring adaptive switching thresholds for better stage control, and extending the framework to real-time applications with efficient inference. Additionally, incorporating multi-view or multi-modal cues could further improve robustness and scene understanding, broadening the scope to more complex dynamic environments.
AI Executive Summary
Generating photorealistic novel views from limited input remains a core challenge in computer vision, especially when geometric priors such as point clouds are sparse or inaccurate. Traditional methods often struggle with error propagation and signal conflicts, leading to structural distortions and artifacts. MoCam introduces a novel solution by embedding a structured denoising dynamic within a latent diffusion framework, which decouples geometric and appearance cues over the denoising process. This stage-wise conditioning strategy begins with geometric scaffold guidance to establish coarse scene structure, then transitions to source appearance cues for detail refinement. This approach effectively prevents geometric errors from contaminating high-fidelity synthesis, resulting in robust, coherent, and photorealistic outputs even under severe geometric degradation. Extensive experiments on datasets like OpenVid-1M demonstrate that MoCam outperforms existing state-of-the-art methods in both static and dynamic view synthesis tasks, achieving lower FVD scores, better pose accuracy, and higher perceptual quality metrics. Its ability to handle sparse, hole-ridden point clouds and large viewpoint changes marks a significant step forward in controllable scene synthesis, with promising applications in virtual production, immersive media, and AR/VR. Looking ahead, further improvements in depth estimation, adaptive conditioning, and real-time inference will broaden its practical impact, paving the way for more intelligent, robust, and versatile scene generation systems.
Deep Analysis
Background
新视角合成作为计算机视觉的核心任务，经历了从基于几何重建到深度学习生成模型的演变。NeRF等技术通过体积渲染实现高质量场景重建，但对多视角数据依赖大，难以应对动态场景。点云和深度图的稀疏、畸变问题限制了其应用范围。近年来，扩散模型在图像生成中表现优异，被引入到3D/4D场景中，推动了单图和视频的高质量生成，但仍面临几何误差传播和信号冲突的挑战。现有方法多采用静态条件融合，难以在大视角变化中保持稳定。MoCam基于潜空间扩散模型，结合点云逆投影和阶段性条件切换，试图解决这一难题，推动生成技术向更鲁棒、更真实的方向发展。
Core Problem
核心问题在于几何先验（如点云）在大视角变化时变得稀疏、畸变，导致误差累积，影响生成质量。传统方法在融合几何和外观信息时，静态融合引发信号冲突，导致结构崩塌和细节丢失。如何在保证几何一致性的同时，提升外观细节的真实性，成为亟待解决的难题。此外，单一阶段条件难以兼顾全局结构和局部细节，导致生成效果不稳定。
Innovation
MoCam的创新点在于引入分阶段条件切换机制：• 在早期阶段，仅利用几何 scaffold 作为引导，建立粗略结构，容忍孔洞和畸变；• 在后期切换到源视频外观信息，主动修正几何误差，丰富细节。此机制避免了静态融合中的信号冲突，提升了几何和外观的解耦能力。结合潜空间扩散模型，通过逆投影和条件调控，实现多模态信息的动态调配。模型在稀疏点云和畸变场景中表现出优异鲁棒性，显著优于传统方法。
Methodology
- �� 构建点云 scaffold：利用深度估计和逆投影，将单视图或视频帧转换为动态点云。• 生成粗糙 scaffold 视频：在目标轨迹上渲染点云，获得空间对齐的初步结构。• 潜空间扩散模型：基于预训练的潜空间扩散架构，学习逆过程中的噪声预测。• 条件切换策略：在扩散的不同阶段，使用阈值Tswitch，将条件从几何 scaffold切换到源视频外观。• 早期阶段：仅用cren条件，建立全局结构。• 后期阶段：切换到csrc条件，修正细节和几何误差。• 最终解码：利用解码器生成目标视频，保证结构和细节的统一。
Experiments
在OpenVid-1M和自采集数据集上，训练模型并进行多场景测试。采用FVD、CLIP-V、姿态误差等指标，评估几何一致性和感知质量。对比GEN3C、TrajCrafter、ReCam等方法，进行消融实验验证条件切换的有效性。通过不同视角轨迹，测试模型在孔洞、畸变和大视角变化下的鲁棒性。实验结果显示，MoCam在多个指标上优于对比方法，尤其在点云孔洞严重时仍保持高质量输出。
Results
MoCam在点云孔洞和畸变场景中，FVD指标降低至255.16，优于GEN3C的289.37和TrajCrafter的313.65。单视图3D重建中，姿态误差显著低于对比方法，细节丰富，结构准确。在动态视频重摄中，背景和主体一致性指标优异，感知质量指标提升20%以上。消融实验验证了阶段性条件切换对性能的关键作用，证明了模型在大视角变化中的鲁棒性。整体表现表明，MoCam有效缓解了几何误差传播，提升了生成的稳定性和真实性。
Applications
该技术适用于虚拟现实、影视特效、虚拟试衣和内容创作等场景，尤其在缺乏完整几何信息的条件下实现高质量新视角生成。只需少量输入（单图或短视频）即可生成多视角内容，极大降低了数据采集成本。未来还可结合实时推理，支持交互式内容生成，为行业带来革命性变革。
Limitations & Outlook
模型对深度估计的依赖较大，深度误差可能影响最终效果。大视角变化或复杂动态场景仍存在畸变风险，训练成本高，需大量配对数据。未来需优化深度估计、提升模型泛化能力，并降低计算成本，以实现更广泛的应用。
Plain Language Accessible to non-experts
想象你在拼一幅拼图，但拼图碎片不完整或有破损。你先用大块拼出整体轮廓，确保结构合理。然后，再用细碎的部分填补空白，修正之前的错误，让拼图变得完整。这就像MoCam的方法：先用粗略的几何信息搭建场景的骨架，确保整体布局正确；接着，用外观信息丰富细节，修正误差，最终呈现逼真的场景。这个过程避免了在一开始就堆砌所有细节，导致错误难以修正。它像是在逐步完善一件艺术品，每一步都确保整体协调，最后得到既有结构又细腻的作品。
ELI14 Explained like you're 14
想象你在玩拼图游戏，但拼图的碎片有点破损或不完整。你先用大块的拼图拼出大致的轮廓，确保整体形状正确；然后，再用细碎的碎片填补空缺，修正之前的错误，让拼图变得完整又漂亮。MoCam的方法也是这样：它先用粗略的几何信息搭建场景的骨架，确保整体结构没问题；接着，用细节丰富的外观信息修正和完善，让场景看起来更真实。这种逐步修正的方法，避免了在一开始就把所有细节堆在一起，导致错误难以修正。最终，场景既有合理的结构，又充满细节，就像一幅生动的画一样。
Glossary
扩散模型 (Diffusion Model)
一种生成模型，通过逐步去噪实现高质量图像或场景合成。技术上利用逆扩散过程预测噪声，生成逼真内容。
论文中用于潜空间的高效生成与条件引导。
潜空间 (Latent Space)
经过编码的压缩表示空间，用于高效存储和操作复杂场景信息。模型在此空间中进行生成和修正。
MoCam利用潜空间扩散实现多模态条件调控。
点云 (Point Cloud)
由空间中的点组成的三维数据，用于表示场景几何结构。稀疏或畸变会影响几何准确性。
作为几何 scaffold，用于引导场景结构。
条件切换 (Conditional Switching)
在生成过程中根据阶段调整引导条件，避免信号冲突，提升生成稳定性。
MoCam的核心创新机制。
逆投影 (Inverse Projection)
将二维图像或深度信息转换为三维点云的过程，用于构建场景几何。
生成粗略几何 scaffold。
Open Questions Unanswered questions from this research
- 1 如何进一步减少深度估计误差对生成效果的影响，提升模型在极端场景下的鲁棒性。
- 2 探索多模态信息融合机制，增强几何与外观的协同优化能力。
Abstract
Generative novel view synthesis faces a fundamental dilemma: geometric priors provide spatial alignment but become sparse and inaccurate under view changes, while appearance priors offer visual fidelity but lack geometric correspondence. Existing methods either propagate geometric errors throughout generation or suffer from signal conflicts when fusing both statically. We introduce MoCam, which employs structured denoising dynamics to orchestrate a coordinated progression from geometry to appearance within the diffusion process. MoCam first leverages geometric priors in early stages to anchor coarse structures and tolerate their incompleteness, then switches to appearance priors in later stages to actively correct geometric errors and refine details. This design naturally unifies static and dynamic view synthesis by temporally decoupling geometric alignment and appearance refinement within the diffusion process. Experiments demonstrate that MoCam significantly outperforms prior methods, particularly when point clouds contain severe holes or distortions, achieving robust geometry-appearance disentanglement.