BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks
BlendedMVS dataset combines 3D reconstruction and image blending, improving deep stereo model generalization.
Key Findings
Methodology
This work constructs a large-scale dataset by reconstructing textured 3D models from well-selected scenes using Altizure. These models are rendered into multi-view color images and depth maps. To simulate realistic lighting, the rendered images are blended with original input images through frequency domain filtering, combining high-frequency details with low-frequency ambient lighting. This process automates data generation, covering over 17,000 high-res images across diverse scenes like cities, architecture, sculptures, and small objects. The dataset enables training of models such as MVSNet, R-MVSNet, and Point-MVSNet, which demonstrate superior generalization on unseen scenes compared to models trained on DTU, ETH3D, or MegaDepth datasets.
Key Results
- Models trained on BlendedMVS show a 15% reduction in average End Point Error (EPE) and higher f-scores (~0.532) on Tanks and Temples, outperforming counterparts trained on traditional datasets. Ablation studies confirm that frequency domain blending of environment lighting significantly enhances cross-scene performance. The models exhibit robustness in urban and sculpture scenes, with denser point clouds and lower errors, confirming the dataset’s effectiveness.
- The fusion of high-frequency details from rendered images with low-frequency ambient lighting from input images yields more realistic training samples, leading to better generalization. Data augmentation strategies further improve robustness, reducing overfitting. Cross-dataset validation demonstrates that models trained on BlendedMVS outperform those trained on DTU, ETH3D, and MegaDepth in both depth accuracy and point cloud completeness.
- Quantitative evaluations reveal that the proposed dataset boosts the average F-score in point cloud reconstruction from 0.475 (DTU) to 0.532, with consistent improvements across diverse scenes. The ablation results highlight the importance of environment lighting in training, and the large scene diversity ensures broad applicability.
Significance
This research addresses the critical bottleneck of limited training data in deep multi-view stereo, providing a scalable, low-cost solution for generating diverse, realistic datasets. The automated pipeline reduces reliance on expensive hardware, enabling rapid expansion of training resources. The resulting models demonstrate enhanced generalization, crucial for real-world applications like urban mapping, cultural heritage digitization, and robotics. By sharing the dataset openly, the work fosters community collaboration, accelerating innovation in 3D reconstruction and scene understanding. It paves the way for more robust, adaptable deep learning models capable of handling complex, real-world environments.
Technical Contribution
The paper introduces a novel pipeline combining 3D textured model reconstruction with frequency domain blending to generate high-quality training data. This method automates large-scale dataset creation, integrating environment lighting and scene details effectively. It leverages differentiable rendering and frequency filtering to produce realistic, diverse samples, surpassing traditional synthetic or real data limitations. The approach enables training of multiple state-of-the-art deep MVS networks with improved cross-scene performance, demonstrating the importance of data diversity and realism. The pipeline’s modularity and low cost make it highly scalable and adaptable for future extensions.
Novelty
This work is the first to systematically integrate 3D textured model rendering with frequency domain blending to produce large-scale, realistic training datasets for deep multi-view stereo. Unlike prior datasets relying solely on real scans or simple synthetic images, BlendedMVS combines the advantages of both, capturing environmental lighting and scene details efficiently. Its automation and scene diversity set a new standard for training data quality and scale, significantly advancing the field’s ability to develop models with strong generalization capabilities.
Limitations
- The quality of training samples depends on the accuracy of 3D reconstruction; defects or incomplete meshes may introduce noise, affecting model training. In complex scenes with occlusions or reflective materials, reconstruction errors can be significant.
- The frequency domain blending assumes low-frequency ambient lighting and high-frequency details, which may not fully capture complex lighting effects like shadows or specular reflections, limiting realism in certain scenarios.
- Current approach primarily targets static scenes; dynamic environments with moving objects or changing lighting conditions are not yet addressed, requiring further methodological development.
Future Work
Future directions include integrating more sophisticated lighting models, such as physically-based rendering, to enhance realism. Expanding the dataset to include dynamic scenes and temporal information will improve models’ robustness in real-world applications. Incorporating self-supervised learning techniques could reduce dependence on reconstruction quality. Additionally, developing multi-modal datasets combining visual, LiDAR, and semantic data will further advance scene understanding and reconstruction capabilities.
AI Executive Summary
Deep multi-view stereo (MVS) has become a cornerstone of 3D scene reconstruction, yet its progress is hampered by the scarcity of large-scale, diverse training datasets. Traditional data collection relies on expensive hardware and labor-intensive processes, limiting the scope and generalization of learned models. Addressing this challenge, the authors introduce BlendedMVS, a novel dataset generated through an automated pipeline that combines 3D textured model reconstruction with frequency domain blending. This approach synthesizes realistic multi-view images and depth maps, incorporating environmental lighting and scene details at a fraction of the cost of physical scanning.
The pipeline begins with high-quality 3D reconstruction of well-selected scenes using the Altizure platform, producing textured meshes. These meshes are rendered from multiple viewpoints to generate color images and depth maps. To simulate realistic lighting conditions, the rendered images are blended with original input images via frequency domain filtering, capturing both scene details and ambient illumination. This process results in a large-scale dataset of over 17,000 high-resolution images covering cityscapes, architecture, sculptures, and small objects.
Extensive experiments demonstrate that models trained on BlendedMVS outperform those trained on traditional datasets like DTU, ETH3D, and MegaDepth, especially in cross-scene generalization tasks. Quantitative metrics show a 15% reduction in depth estimation errors and higher point cloud reconstruction scores. Ablation studies confirm the effectiveness of frequency domain blending and data augmentation strategies, which significantly improve robustness and realism.
This work marks a significant step towards scalable, low-cost data generation for deep learning in 3D reconstruction. By providing a rich, diverse training resource, it enables the development of models capable of handling complex, real-world environments. The open sharing of the dataset fosters community collaboration, accelerating innovation in scene understanding, virtual reality, and cultural heritage preservation. Future efforts will focus on incorporating dynamic scenes, advanced lighting models, and multi-modal data to further enhance model performance and applicability.
Deep Analysis
Background
多视Stereo（MVS）作为三维重建的核心技术，经历了从传统几何算法到深度学习的快速发展。早期方法如PatchMatch和多视几何算法在某些场景中取得了良好效果，但在纹理缺失或非Lambertian表面表现不足。近年来，深度学习模型如MVSNet、R-MVSNet和Point-MVSNet引入端到端训练，显著提升了重建质量。然而，这些模型依赖的训练数据规模有限，主要来自DTU、ETH3D等小规模场景，限制了其泛化能力。合成数据的应用逐渐兴起，但缺乏真实环境光照和丰富细节，影响模型在实际场景中的表现。本文提出的BlendedMVS结合3D重建和图像融合技术，旨在解决这一瓶颈，提供多样、真实的训练样本。
Core Problem
现有深度学习模型在未见场景中的表现仍有限，主要受限于训练数据的规模和多样性。传统点云采集设备昂贵且繁琐，难以实现大规模、多场景的覆盖。公开数据集如DTU和ETH3D场景单一，缺乏环境光照和细节丰富的样本，导致模型泛化能力不足。此外，纯合成渲染的样本缺乏真实环境光照信息，难以在复杂场景中表现出色。如何低成本、高效率地生成具有环境光照和丰富细节的多样化训练数据，成为亟待解决的核心问题。
Innovation
本研究的创新点包括：1）提出结合3D纹理模型渲染与频域融合的自动化数据生成流程，显著降低成本，支持大规模多场景训练；2）引入频域滤波技术，将渲染图的细节（高频）与输入图的环境光照（低频）融合，增强样本真实性；3）实现多视角、多场景的自动合成，提升模型在未见场景中的泛化能力。该方法突破了传统依赖昂贵设备采集点云的限制，为深度学习多视Stereo提供了丰富的训练资源。
Methodology
- �� 采集场景：利用Altizure平台进行3D纹理模型重建，获得高质量纹理网格。• 渲染：在不同视角下将模型渲染成彩色图像和深度图。• 融合：采用频域滤波，将渲染图的细节（高频）与输入图的环境光照（低频）结合，生成逼真的训练样本。• 数据增强：应用随机亮度、对比度和运动模糊，提升模型鲁棒性。• 数据集构建：筛选113个多样场景，生成超过17,000张高分辨率图像，场景涵盖城市、建筑、雕塑等。• 模型训练：用MVSNet、R-MVSNet和Point-MVSNet在该数据集上进行训练，验证其泛化能力。
Experiments
设计了多场景、多模型训练验证方案，比较在DTU、ETH3D、Tanks and Temples等公开数据集上的深度误差和点云指标。采用端点误差（EPE）、像素误差比例和f-score作为指标。通过消融实验验证频域融合的效果，分析不同融合策略对性能的影响。跨场景验证确保模型在未见场景中表现优异。参数设置包括深度范围、输入分辨率和数据增强策略，确保实验公平性。
Results
训练于BlendedMVS的数据显著降低深度估计误差，平均EPE减少15%，f-score提升至0.532，优于基于DTU和MegaDepth的模型。频域融合显著增强模型跨场景能力，验证中城市和雕塑场景表现尤佳，点云密度更高，误差更低。消融实验确认环境光照融合的重要性，数据多样性确保模型适应复杂环境。
Applications
该数据集适用于三维重建、虚拟现实、文化遗产数字化等。工业界可利用其提升复杂环境下的模型适应性，文化保护机构借助其进行数字存档。未来结合多模态信息，将推动无人机、机器人等智能系统的环境感知与导航能力提升。
Limitations & Outlook
依赖于3D重建的质量，模型缺陷或不完整会影响训练效果。融合方法在极端光照或复杂材质场景中仍有不足。主要面向静态场景，动态环境尚未覆盖，未来需引入运动和时间信息以增强模型鲁棒性。
Plain Language Accessible to non-experts
想象你在拼一幅大拼图，你有很多碎片（照片），每个碎片都来自不同角度。以前，要拼出完整的场景，得用昂贵的扫描仪和繁琐的测量工具，非常费时费钱。而这篇论文像发明了一种魔法拼图方法，只用普通照片，经过特殊处理，就能拼出逼真的3D场景。它先用电脑把碎片变成3D模型，然后用特殊的滤镜把细节和光线效果融合在一起，让拼图看起来既真实又丰富。这样，不用花大钱就能得到各种场景的拼图，帮助电脑更聪明地理解世界。这就像用魔法拼图，让机器学习变得更强大、更灵活，能在城市、雕塑、建筑等各种场景中都表现出色。
ELI14 Explained like you're 14
想象你在玩一个超级酷的拼图游戏，你有很多照片，从不同角度拍摄一座雕塑或者一座城市。以前，要让电脑知道这些照片背后的3D模型，得用昂贵的设备和很长时间去扫描，非常麻烦。而这篇论文就像发明了一种神奇的拼图魔法，只需要用普通的照片，经过一些聪明的处理，就能拼出非常逼真的3D模型。它会先用电脑把这些照片变成一个完整的3D模型，然后用特殊的滤镜，把模型的细节和光线效果融合在一起，让模型看起来和真实一样。这样，电脑就能更快、更好地理解复杂的场景，比如城市街道或者雕塑。这个方法让我们不用花很多钱和时间，就能得到很多漂亮的3D场景，帮助机器人、游戏和文化保护变得更智能、更有趣。
Glossary
3D Reconstruction (三维重建)
利用多视角图像和算法生成场景的三维模型，广泛应用于虚拟现实和测绘。
本文通过3D重建获得纹理模型，为渲染和融合提供基础。
Frequency Domain Filtering (频域滤波)
在频率空间对图像信号进行处理，用于提取或抑制特定频率成分，增强图像细节或环境光照。
融合环境光照和细节信息的关键技术。
Deep Learning Models (深度学习模型)
基于神经网络的算法，用于自动学习特征和进行复杂任务，如深度估计。
训练模型包括MVSNet、R-MVSNet和Point-MVSNet。
Synthetic Dataset (合成数据集)
通过计算机模拟生成的训练数据，成本低、规模大，便于模型泛化。
本文构建了规模超过17,000张图像的合成数据集。
Generalization Ability (泛化能力)
模型在未见过的场景或数据上的表现能力，是衡量模型实用性的关键指标。
本研究通过多场景数据提升模型的泛化能力。
Open Questions Unanswered questions from this research
- 1 如何进一步提升合成数据的真实性，尤其在复杂光照和材质条件下的表现仍有待研究。
- 2 动态场景的多视Stereo训练数据缺乏，未来应考虑引入运动和时间信息以增强模型适应性。
Abstract
While deep learning has recently achieved great success on multi-view stereo (MVS), limited training data makes the trained model hard to be generalized to unseen scenarios. Compared with other computer vision tasks, it is rather difficult to collect a large-scale MVS dataset as it requires expensive active scanners and labor-intensive process to obtain ground truth 3D structures. In this paper, we introduce BlendedMVS, a novel large-scale dataset, to provide sufficient training ground truth for learning-based MVS. To create the dataset, we apply a 3D reconstruction pipeline to recover high-quality textured meshes from images of well-selected scenes. Then, we render these mesh models to color images and depth maps. To introduce the ambient lighting information during training, the rendered color images are further blended with the input images to generate the training input. Our dataset contains over 17k high-resolution images covering a variety of scenes, including cities, architectures, sculptures and small objects. Extensive experiments demonstrate that BlendedMVS endows the trained model with significantly better generalization ability compared with other MVS datasets. The dataset and pretrained models are available at \url{https://github.com/YoYo000/BlendedMVS}.