Video Generative Models as Geometry Learner
Utilizes pretrained video diffusion models for joint monocular depth and normal estimation via next-frame prediction, reducing data needs significantly.
Key Findings
Methodology
This paper introduces GeoNeXt, which adapts pretrained video diffusion models to estimate geometry by framing it as a next-frame prediction task. It replicates input images into depth and normal slots, fine-tunes a lightweight U-Net in the noise space, and leverages Latent Diffusion with VAE encoding to reduce computation. The model jointly generates image and geometry, ensuring structural consistency with minimal labeled data. During inference, iterative denoising produces depth and normal maps aligned with the input image, utilizing the model’s temporal priors for cross-modal coherence.
Key Results
- GeoNeXt outperforms task-specific and unified generative models on zero-shot monocular depth and normal estimation across datasets like ETH3D, KITTI, and ScanNet. For example, on ETH3D, it achieves an AbsRel of 5.6, surpassing many supervised methods trained on orders of magnitude more data. It demonstrates strong generalization with significantly less training data, often using less than 1% of the data required by large discriminative models.
Significance
This approach addresses the long-standing challenge of high data dependency in monocular geometry estimation, offering a data-efficient, scalable solution. By leveraging the intrinsic priors of video models, it enables high-quality 3D scene understanding with minimal supervision, impacting autonomous driving, AR/VR, and 3D reconstruction industries. It also opens new avenues for cross-modal learning and few-shot adaptation.
Technical Contribution
The key innovation lies in repurposing video diffusion models for geometric tasks, formulating joint image-geometry generation as a next-frame prediction problem. The method involves minimal architectural modifications, only fine-tuning the U-Net, and employs latent space diffusion with VAE to improve efficiency. This unified framework enhances cross-modal consistency and reduces training data requirements, setting a new paradigm for generative geometric estimation.
Novelty
This is the first systematic application of pretrained video diffusion models to monocular depth and normal estimation, introducing a joint, multi-modal generation framework. Unlike prior work that trains separate or architectural-modified models, GeoNeXt exploits the temporal priors of video models for coherent, data-efficient geometry inference, representing a significant step forward in generative 3D vision.
Limitations
- The model struggles with extremely complex scenes, especially under severe occlusion or lighting variations. Its reliance on pretrained video priors means domain shifts can affect performance. The iterative denoising process limits real-time deployment. Further work is needed to enhance detail recovery and inference speed.
Future Work
Future directions include integrating multi-modal cues, improving real-time inference, and extending to more complex scenes. Combining self-supervised learning could further reduce data dependence. Exploring larger or more diverse video models may enhance robustness and detail fidelity, broadening practical deployment.
AI Executive Summary
Single-view 3D scene understanding remains a fundamental yet challenging problem in computer vision. Traditional methods rely heavily on large annotated datasets, which are costly and limit scalability. Recent advances in generative models, especially diffusion-based frameworks, have shown promise in image synthesis, but their application to geometric estimation has been limited.
This paper introduces GeoNeXt, a novel approach that leverages pretrained video diffusion models for joint monocular depth and surface normal estimation. By formulating the task as a next-frame prediction problem, the method exploits the rich temporal priors learned from large-scale video datasets. The core idea involves replicating the input image into depth and normal slots, then jointly generating these along a denoising trajectory, ensuring structural coherence. Only the U-Net component is fine-tuned, significantly reducing training costs while maintaining high performance.
The approach employs Latent Diffusion with VAE encoding, enabling efficient computation and high-resolution synthesis. Extensive experiments across multiple datasets demonstrate that GeoNeXt surpasses existing task-specific and unified generative models, often with less than 1% of the training data used by large discriminative models. For instance, on ETH3D, it achieves an AbsRel of 5.6, outperforming many supervised methods.
This work marks a significant step toward data-efficient, scalable 3D scene understanding. Its ability to generalize zero-shot across diverse datasets indicates strong potential for real-world applications such as autonomous navigation, AR/VR, and 3D content creation. Future work will focus on improving detail fidelity, inference speed, and extending to more complex, dynamic scenes, further broadening the impact of generative geometric estimation.
Deep Analysis
Background
Monocular 3D geometry estimation一直是计算机视觉中的核心难题，早期方法依赖大量标注数据，采用深度学习模型如DPT、MiDaS等。近年来，预训练模型如Stable Diffusion、Text2Image等在生成任务中表现出色，但其在几何估计中的应用仍有限。视频扩散模型如Stable Video Diffusion提供了丰富的时序先验，为几何任务带来新机遇。尽管如此，如何高效利用视频模型的潜力，解决几何估计中的一致性和细节捕获问题，仍是研究热点。
Core Problem
单目深度和法线估计面临数据依赖大、泛化差、细节不足等挑战。现有方法多需大量标注，训练成本高，且在复杂场景中表现不佳。如何在少量标注或零样本条件下实现高精度估计，成为行业难题。视频模型的潜在能力未被充分挖掘，如何迁移其时序先验到几何任务，是核心难点。
Innovation
提出GeoNeXt，将视频扩散模型迁移至几何估计，创新点包括：
- �� 将几何信息作为下一帧预测任务，利用视频模型的时序先验增强结构一致性；
- �� 仅微调U-Net部分，极大降低训练成本；
- ��结合Latent Diffusion和VAE技术，提高推理效率；
- ��实现跨模态的联合建模，提升数据利用率和泛化能力。
Methodology
- �� 输入：单幅RGB图像，编码为潜在空间中的特征；
- �� 复制图像到深度和法线槽，作为条件输入；
- �� 在噪声空间中逐步去噪，生成深度和法线的潜在表示；
- �� 采用Latent Diffusion机制，结合VAE编码，减少计算量；
- �� 训练：最小化噪声预测误差，利用少量RGB-深度-法线三元组；
- �� 推理：初始化高斯噪声，逐步去噪，输出几何图。
- �� 关键在于联合生成图像与几何信息，确保一致性和细节。
Experiments
采用Hypersim和Virtual KITTI两个合成数据集进行训练，验证在NYUv2、KITTI、ScanNet等真实场景数据上的零样本估计能力。对比基线包括判别模型和单任务生成模型，指标为AbsRel、δ1等。通过不同训练策略和参数调优，验证模型的鲁棒性和泛化能力。多场景测试显示，GeoNeXt在数据量极少的情况下，仍能实现优异性能。
Results
在ETH3D上，AbsRel误差为5.6，优于多数对比方法。KITTI和ScanNet上，误差指标也优于任务专用模型。模型在细节重建方面表现出色，尤其在复杂边界和细微结构的捕获上优于其他生成模型。多数据集验证证明其泛化能力强，且训练时间和数据需求远低于传统方法。
Applications
可应用于自动驾驶、虚拟现实、三维重建等场景，依赖少量标注数据即可实现高质量几何估计。适合资源有限的设备和实时系统，为工业界提供低成本高效的解决方案。未来可结合自监督学习，进一步提升模型的适应性和鲁棒性。
Limitations & Outlook
模型在极端光照和遮挡条件下表现仍有限，细节捕获在复杂场景中不足。微调依赖预训练模型，迁移到新领域需额外调优。推理速度受多次去噪限制，实时应用仍需优化。未来需增强模型的细节表达和速度，扩大适用范围。
Plain Language Accessible to non-experts
想象你在厨房做饭，准备一道复杂的菜肴。传统方法需要你事先知道每个食材的详细信息，比如每个蔬菜的重量和切法，才能做出好菜。这就像用大量标注数据训练模型，成本高，效率低。而新方法像是用一种聪明的厨具，它能根据你厨房的环境，自动猜测每个食材的状态和位置，只需少量提示，就能做出美味的菜。它学习了厨房的整体规律，能在没有详细说明的情况下，快速理解和完成任务。这就像GeoNeXt利用视频模型的“厨房经验”，在少量数据下，准确估算场景的深度和表面法线，帮你轻松搞定复杂的三维场景。
Abstract
Recent generative approaches to geometry estimation adapt pretrained image diffusion models and treat the task as image-conditioned generation. Leveraging off-the-shelf image diffusion models, they either (i) train task-specific geometry models (for depth and surface normal estimation) independently, losing the opportunity of exploring the intrinsic correlation of these geometric targets, or (ii) jointly fine-tune modified image diffusion backbones (e.g., altered self-attention), which typically demands substantial labeled data. To overcome these limitations in a principled fashion, we repurpose pretrained video generative models as a unified and data-efficient framework for geometry estimation, formulated innovatively as a next-frames prediction task. Our method, GeoNeXt, inherits naturally structured knowledge and richer priors from the video model, while further adapting them for joint modeling of images and geometry targets (image <-> geometry), enabling more data efficient and effective learning of geometry. Extensive experiments validate our method for zero-shot monocular depth and surface normal estimation across diverse datasets, outperforming both previous task-specific and unified generative competitors while using substantially less training data. Notably, our method rivals discriminative state-of-the-art approaches trained on over 100x more data and even standouts on several benchmarks.