ICDepth: Taming Video Diffusion Models for Video Depth Estimation via In-Context Conditioning
ICDepth uses In-Context Conditioning and SAND-Attention for video depth estimation, achieving 6-13x data efficiency improvement.
Key Findings
Methodology
ICDepth leverages pre-trained text-to-video diffusion models with In-Context Conditioning (ICC), treating RGB and depth features as unified sequences. Key components include SAND-Attention for noise decoupling and SRFM for semantic and resolution priors.
Key Results
- On Sintel, AbsRel reduced by 16%, δ1 improved by 10.1%, outperforming all baselines.
- Trained on only 0.8M frames, achieving 6-13x less data usage while excelling on ScanNet and KITTI benchmarks.
- Under low-light conditions, δ1 degradation was only 4%, demonstrating strong robustness.
Significance
This work addresses challenges in spatiotemporal consistency and geometric precision for video depth estimation while drastically reducing data requirements. It has significant implications for AR/VR, autonomous driving, and 3D reconstruction.
Technical Contribution
Introduced SAND-Attention and SRFM modules to tackle noise contamination and geometric ambiguity, respectively. First application of ICC framework to video depth estimation, enhancing spatiotemporal consistency and precision.
Novelty
ICDepth is the first to adapt pre-trained text-to-video diffusion models for depth estimation, leveraging ICC for cross-modal interaction and innovative modules to address generative model limitations.
Limitations
- Performance on highly dynamic scenes still needs improvement.
- Training data is predominantly synthetic, requiring further validation on real-world datasets.
- Inference time is relatively long, limiting real-time applications.
Future Work
Future research could explore faster inference techniques, extend to real-world datasets, and optimize for extreme dynamic scenarios.
AI Executive Summary
Monocular video depth estimation demands spatiotemporal consistency, geometric precision, and generalization across diverse scenarios, yet existing methods struggle to achieve all three. ICDepth introduces a novel framework that adapts pre-trained text-to-video diffusion models for depth estimation using In-Context Conditioning (ICC), alongside SAND-Attention and SRFM modules to address noise contamination and geometric ambiguity.
Experiments demonstrate ICDepth's superior performance on benchmarks like Sintel and KITTI, with AbsRel reduced by 16% and δ1 improved by 10.1%. It achieves strong robustness under low-light conditions and requires only 0.8M frames for training, significantly reducing data demands.
Despite limitations in dynamic scenes and inference speed, ICDepth sets a new direction for video depth estimation, with impactful applications in AR/VR, autonomous driving, and beyond.
Deep Analysis
Background
Video depth estimation is a critical task in 3D computer vision, enabling applications like AR/VR, autonomous driving, and 3D reconstruction. Prior methods include discriminative models with high geometric accuracy but poor temporal consistency, and generative models requiring extensive data while lacking precision.
Core Problem
Existing approaches fail to balance spatiotemporal consistency, geometric accuracy, and data efficiency. Discriminative models have limited temporal modeling, while generative models demand over 10M samples and lack geometric precision.
Innovation
ICDepth uses ICC to unify RGB and depth features into a single sequence, enabling cross-modal interaction. It introduces SAND-Attention to address noise contamination and SRFM to inject semantic and resolution priors for enhanced precision.
Methodology
- �� Built on pre-trained text-to-video diffusion model Wan2.1.
- �� ICC framework combines RGB and depth features into unified sequences.
- �� SAND-Attention ensures noise decoupling and spatiotemporal alignment.
- �� SRFM module incorporates DINOv2 semantic and resolution priors for improved precision.
Experiments
Evaluated on Sintel, KITTI, and ScanNet datasets using metrics like AbsRel and δ1. Training data includes VKITTI, TartanAir, and OmniWorld, totaling 0.8M frames. Zero-shot tests and robustness under low-light conditions were conducted.
Results
ICDepth achieved 16% AbsRel reduction and 10.1% δ1 improvement on Sintel; excelled on ScanNet and KITTI with 6-13x less data; demonstrated robustness with only 4% δ1 degradation under low-light conditions.
Applications
Direct applications include real-time depth estimation for AR/VR, environmental perception in autonomous driving, and depth generation for 3D animation.
Limitations & Outlook
Challenges include performance in highly dynamic scenes, reliance on synthetic data, and long inference time limiting real-time use.
Plain Language Accessible to non-experts
Imagine filming a video, and ICDepth acts like a smart assistant that can measure the distance of every object in the scene. It's like using a ruler, but smarter—it keeps measurements stable across frames and adjusts for challenging conditions like nighttime or foggy weather.
ELI14 Explained like you're 14
Think of playing a video game with lots of characters and scenes. ICDepth is like a super tool that tells you how far each character or object is. It remembers changes between frames, so the results don't jump around. Even in dark or foggy levels, it works like magic! Cool, right?
Glossary
Diffusion Model
A generative model that creates data by iteratively denoising random noise.
Used for generating depth maps in video estimation.
Temporal Consistency
Ensures depth estimation remains stable across video frames.
Addresses flickering issues in depth maps.
SAND-Attention
A mechanism to decouple noise and align spatiotemporal features.
Key module for ICDepth's precision.
DINOv2
A pre-trained vision model providing semantic priors.
Enhances geometric precision in depth estimation.
In-Context Conditioning (ICC)
A method to unify RGB and depth features into a single sequence.
Enables cross-modal interaction for depth estimation.
Open Questions Unanswered questions from this research
- 1 How to improve model generalization to real-world scenarios?
- 2 How to optimize inference speed for real-time applications?
Applications
Immediate Applications
AR/VR Depth Estimation
Real-time depth map generation for augmented reality devices to enhance user experience.
Autonomous Driving
Used for environmental perception and obstacle detection to improve driving safety.
Long-term Vision
Dynamic Scene Depth Estimation
Optimizing models for complex dynamic scenarios to enable broader applications.
Abstract
Monocular video depth estimation requires temporal consistency, geometric accuracy, and generalization across diverse scenarios, yet existing methods struggle to achieve all three simultaneously. Discriminative models excel at per-frame accuracy but suffer from temporal drift due to limited context windows, while generative methods improve consistency and generalization at the cost of extensive training data (10M+ samples) and lack of geometric precision. In response to these issues, we introduce \textbf{ICDepth}, a framework that adapts pre-trained text-to-video diffusion transformers for video depth estimation via In-Context Conditioning (ICC), leveraging their rich spatial-temporal priors. To address key challenges in transferring ICC from generation to dense prediction, we propose: (1)~\textbf{SAND-Attention}, which ensures precise spatial-temporal alignment via shared RoPE and enforces unidirectional attention to prevent noise contamination; (2)~\textbf{SRFM}, which injects DINOv2 semantic and resolution priors to enhance geometric precision. ICDepth achieves state-of-the-art results on multiple benchmarks with remarkable data efficiency, trained on only 0.8M frames ($6$--$13\times$ less than competing generative methods), while demonstrating strong zero-shot generalization to diverse domains.