MegaDepth: Learning Single-View Depth Prediction from Internet Photos
Using internet multi-view images with SfM and MVS to create MegaDepth, greatly enhancing generalization in single-view depth prediction.
Key Findings
Methodology
The paper combines structure-from-motion (SfM) and multi-view stereo (MVS) to automatically generate dense depth maps from vast internet photo collections. By refining raw MVS outputs with depth filtering and semantic segmentation, noisy and dynamic regions are excluded. The training employs a scale-invariant loss (Lsi), multi-scale gradient matching (Lgrad), and a robust ordinal depth loss (Lord). The resulting MegaDepth dataset contains about 150,000 images covering diverse scenes worldwide, providing rich training data. Models trained solely on MegaDepth demonstrate strong generalization, outperforming prior datasets like Make3D, KITTI, and DIW, even on unseen environments.
Key Results
- On the MD test set, the ResNet-based model achieves a scale-invariant RMSE of 0.104 and reduces depth disagreement rate (SDR) to 25.82%, surpassing models trained on sensor-based datasets. Cross-dataset tests show significant improvements: in Make3D, absolute relative error drops from 0.614 to 0.364; in KITTI, RMS error reduces from 12.15 to 6.68. The model maintains robustness in complex scenes with dynamic objects and sky regions, thanks to depth filtering and semantic enhancement.
- Ablation studies reveal that removing depth filtering or the ordinal loss degrades performance, confirming their importance. The multi-scale gradient loss (Lgrad) sharpens depth discontinuities, while the ordinal loss (Lord) preserves structural details. The model's ability to generalize across datasets demonstrates the effectiveness of large-scale internet data for training deep models.
- The approach achieves state-of-the-art results without requiring scene-specific or sensor-based labels, enabling scalable, cost-effective depth estimation for real-world applications like autonomous driving, robotics, and AR/VR. The results validate the hypothesis that diverse, automatically generated data can significantly improve model robustness and accuracy.
Significance
This work addresses the bottleneck of limited, scene-specific depth datasets by leveraging the vast, diverse pool of internet images. It demonstrates that large-scale, automatically reconstructed data can train models with superior generalization, reducing reliance on expensive sensor data. This paradigm shift opens new avenues for scalable 3D scene understanding, crucial for autonomous systems, virtual environments, and robotics. The integration of multi-view geometry with deep learning exemplifies a powerful hybrid approach, pushing the frontier of single-view depth prediction. Furthermore, the techniques for noise filtering, semantic augmentation, and ordinal relation learning contribute valuable tools for future research in 3D vision.
Technical Contribution
The paper introduces a comprehensive pipeline combining SfM and MVS to generate large-scale, high-quality depth datasets from internet images. Key innovations include a depth refinement method that compares depth estimates to reduce background bleeding, semantic segmentation-based filtering to exclude dynamic objects, and automatic ordinal depth labeling to encode relative depth relationships. The loss function integrates scale-invariant terms, multi-scale gradient matching, and robust ordinal constraints, enabling the model to learn detailed, structurally consistent depth maps. This approach significantly enhances the generalization ability of deep models trained on noisy, real-world data, marking a step forward in data-driven 3D scene understanding.
Novelty
This is the first work to systematically utilize internet-based SfM+MVS reconstructions for training deep single-view depth models at scale. Unlike prior datasets limited to indoor or specific scenarios, MegaDepth harnesses the diversity of global landmarks, automatically filtering noise and encoding ordinal relations. The integration of semantic segmentation for filtering and annotation, combined with a novel loss function, distinguishes this work from previous approaches relying on sensor data or sparse annotations. It demonstrates that large, noisy datasets can be effectively used for training robust models, a significant leap in the field.
Limitations
- Despite improvements, MVS still struggles with transparent, reflective, or highly dynamic objects, leading to incomplete or inaccurate depth maps. Noise and outliers persist, especially in complex environments, affecting training quality.
- The reliance on semantic segmentation introduces errors when segmentation fails, potentially filtering out valid data or retaining noisy regions. Extreme lighting or occlusion conditions further challenge robustness.
- Computationally, the pipeline requires extensive processing for large datasets, limiting real-time applicability. Scalability to video sequences or dynamic scenes remains an open challenge, requiring further algorithmic optimization.
Future Work
Future directions include developing more robust multi-view algorithms capable of handling dynamic scenes, integrating temporal information for video-based depth estimation, and reducing computational costs. Exploring self-supervised or weakly supervised learning paradigms could further leverage unlabeled data. Extending the approach to outdoor scenes with varying weather and lighting conditions, and improving semantic segmentation accuracy, will broaden applicability. Additionally, real-time deployment for autonomous systems remains a key goal.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一个工厂里，工人们用不同的工具制作一件产品。每个人都从不同角度观察，记录下各个部分的距离和位置。工厂里的机器人就像是用这些观察结果，自己学习如何判断每个零件离自己有多远。以前，工人们需要用尺子一一测量，但现在，机器人通过观察很多照片，学会了用视觉判断距离，就像你用眼睛看东西一样快。这就像是让机器人变得更聪明，能在不同的工厂里都知道每个零件的位置，而不用每次都用尺子测量。这样，机器人可以更快、更准确地帮忙装配，甚至还能在没有人工帮助的情况下自主工作。
ELI14 Explained like you're 14
你有没有想过，电脑能不能像人一样，用眼睛看东西，然后知道它们离自己有多远？以前，要让电脑知道距离，得用特殊的设备，比如激光雷达，成本很高。现在，科学家们想出了一个聪明的办法：让电脑看很多网上的照片，自己学习怎么判断距离。就像你看一张照片，看到远处的山和近处的树，你会觉得哪个离你更近。电脑也可以通过观察很多不同角度的图片，学会了用“眼睛”判断距离。它不用特殊设备，只靠这些图片，就能知道场景里每个东西的远近。这对自动驾驶汽车、机器人和虚拟现实都特别有用，因为它们都需要知道周围环境的空间结构。未来，这种方法还能让电脑更聪明，能在动态场景中快速判断距离，就像你在玩游戏时，知道哪个敌人在你身后一样酷！
Abstract
Single-view depth prediction is a fundamental problem in computer vision. Recently, deep learning methods have led to significant progress, but such methods are limited by the available training data. Current datasets based on 3D sensors have key limitations, including indoor-only images (NYU), small numbers of training examples (Make3D), and sparse sampling (KITTI). We propose to use multi-view Internet photo collections, a virtually unlimited data source, to generate training data via modern structure-from-motion and multi-view stereo (MVS) methods, and present a large depth dataset called MegaDepth based on this idea. Data derived from MVS comes with its own challenges, including noise and unreconstructable objects. We address these challenges with new data cleaning methods, as well as automatically augmenting our data with ordinal depth relations generated using semantic segmentation. We validate the use of large amounts of Internet data by showing that models trained on MegaDepth exhibit strong generalization-not only to novel scenes, but also to other diverse datasets including Make3D, KITTI, and DIW, even when no images from those datasets are seen during training.