Recurrent MVSNet for High-resolution Multi-view Stereo Depth Inference
Proposes R-MVSNet with convolutional GRU for recursive cost volume regularization, reducing memory by over 4.6×, enabling high-res large-scale 3D reconstruction.
Key Findings
Methodology
This paper introduces R-MVSNet, which replaces traditional 3D CNN regularization with sequential processing using convolutional GRU along the depth axis. The network builds on MVSNet, utilizing differentiable homographies and variance-based cost metrics, but processes the cost volume step-by-step, significantly reducing memory consumption. Multiple stacked GRU layers enable effective aggregation of spatial and depth context, transforming the depth inference into a recurrent classification task trained with cross-entropy loss. Experiments on DTU, Tanks and Temples, and ETH3D datasets demonstrate the method's superior scalability and accuracy, especially in large-scale, high-resolution scenes.
Key Results
- On DTU, R-MVSNet achieved a mean accuracy of 0.385mm and a completeness of 0.459, outperforming original MVSNet. In Tanks and Temples, it ranked third, successfully reconstructing complex outdoor scenes with improved point cloud density. On ETH3D, it maintained competitive performance with a 4.6× memory efficiency gain, supporting wider depth ranges and higher resolutions. Ablation studies confirmed that recurrent regularization surpasses spatial-only or winner-take-all approaches, validating the core innovation.
- The multi-layer stacked GRU architecture effectively regularizes the cost volume with reduced memory, enabling high-resolution reconstructions (up to D=512). The method's scalability was demonstrated by reconstructing large scenes with minimal memory footprint, outperforming prior methods by up to 8× in efficiency. The approach also preserves fine details, with the depth error remaining below 0.42mm even at increased depth samples, showing robustness across diverse scenarios.
Significance
This work addresses the longstanding challenge of high memory consumption in deep learning-based multi-view stereo, enabling large-scale, high-resolution scene reconstruction. By introducing a recursive regularization framework, it extends the applicability of learned MVS methods to complex, real-world environments. The technique's scalability and efficiency open new avenues for industrial applications such as cultural heritage preservation, urban mapping, and autonomous navigation, where detailed 3D models are essential. It also provides a foundation for future research into memory-efficient neural architectures for 3D vision tasks.
Technical Contribution
The core technical innovation lies in replacing the computationally intensive 3D CNN regularization with convolutional GRU units that process the cost volume sequentially along the depth axis. This approach reduces memory complexity from cubic to quadratic relative to scene resolution. The multi-layer stacked GRU enhances the model's capacity to capture spatial and depth context, enabling accurate depth inference across wide ranges. The training employs a classification loss on the probability volume, facilitating sub-pixel accuracy through depth map refinement. The method's design balances efficiency and precision, setting a new standard for large-scale deep learning-based 3D reconstruction.
Novelty
This is the first work to employ convolutional GRU for sequential regularization of the cost volume in multi-view stereo, effectively addressing the memory bottleneck of prior 3D CNN-based methods. Unlike traditional approaches that process the entire volume at once, the recursive scheme allows scalable high-resolution reconstructions. The integration of multi-layer stacked GRU and the end-to-end training framework further distinguishes this work, offering a novel paradigm for large-scale 3D scene understanding.
Limitations
- Despite its scalability, the method's performance can degrade in scenes with severe occlusions or textureless regions, where depth cues are weak. Handling dynamic scenes remains challenging due to reliance on static assumptions. Although memory efficiency is improved, the computational cost for very dense depth sampling at ultra-high resolutions can still be significant. Future work should focus on robustness to occlusion, real-time capabilities, and adaptive sampling strategies.
Future Work
Future directions include integrating multi-scale and attention mechanisms to further improve global consistency and robustness. Developing unsupervised or semi-supervised training regimes could reduce dependence on ground truth data. Enhancing real-time inference capabilities and extending the framework to dynamic scenes will broaden practical applications. Additionally, combining this approach with semantic understanding could enable more intelligent scene reconstruction and analysis.
AI Executive Summary
Multi-view stereo (MVS) has long been a cornerstone of 3D scene reconstruction, yet traditional deep learning approaches like MVSNet face significant limitations when scaling to high-resolution, large-scale environments. The core challenge stems from the cubic growth in memory required to regularize the 3D cost volume using conventional 3D CNNs, which restricts scene size and detail. Addressing this bottleneck, the authors propose R-MVSNet, a novel framework that replaces full-volume 3D convolution with a sequential regularization strategy based on convolutional gated recurrent units (Conv-GRU). This design enables the model to process the cost volume slice-by-slice along the depth axis, dramatically reducing memory consumption by over 4.6× without sacrificing accuracy.
The architecture builds upon the foundation of MVSNet, incorporating differentiable homographies and variance-based cost metrics for initial cost volume construction. The key innovation lies in stacking multiple GRU layers that iteratively update the cost maps, effectively capturing spatial and depth context. The network is trained as a classification problem, predicting depth probabilities with a cross-entropy loss, and refined further with a variational depth refinement step. Extensive experiments on DTU, Tanks and Temples, and ETH3D datasets demonstrate that R-MVSNet achieves state-of-the-art or comparable results, with the added advantage of scalability to scenes with wider depth ranges and higher resolutions.
In practical terms, the method enables high-quality 3D reconstruction of large outdoor scenes, complex architectural environments, and cultural heritage artifacts, which were previously infeasible due to memory constraints. The experimental results show that the approach maintains sub-millimeter accuracy, with a mean error below 0.42mm, while significantly reducing GPU memory usage. Ablation studies confirm the effectiveness of the recurrent regularization over traditional spatial or winner-take-all methods. Despite some limitations in occlusion handling and dynamic scenes, the framework sets a new benchmark for scalable deep learning-based 3D reconstruction, opening pathways for real-world industrial applications and future research into memory-efficient neural architectures.
Deep Dive
Key Concepts
Cost Volume
A 3D representation combining multiple view correspondences to estimate depth; in this paper, regularized sequentially to reduce memory.
Convolutional GRU
A recurrent neural network unit that combines convolutional operations with gating mechanisms, used here for sequential regularization.
Differentiable Homography
A geometric transformation allowing warping of images between views in a neural network, enabling end-to-end training.
Depth Map Refinement
A process to improve initial depth estimates by enforcing multi-view consistency and sub-pixel accuracy.
Memory Efficiency
The ability of the proposed method to process large scenes with reduced GPU memory, crucial for high-res reconstruction.
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在遮挡和纹理缺失场景中的鲁棒性，仍是未来研究的重点。当前方法在极端光照变化和动态场景中的表现有限，需结合多尺度和注意力机制增强模型的适应性。
Abstract
Deep learning has recently demonstrated its excellent performance for multi-view stereo (MVS). However, one major limitation of current learned MVS approaches is the scalability: the memory-consuming cost volume regularization makes the learned MVS hard to be applied to high-resolution scenes. In this paper, we introduce a scalable multi-view stereo framework based on the recurrent neural network. Instead of regularizing the entire 3D cost volume in one go, the proposed Recurrent Multi-view Stereo Network (R-MVSNet) sequentially regularizes the 2D cost maps along the depth direction via the gated recurrent unit (GRU). This reduces dramatically the memory consumption and makes high-resolution reconstruction feasible. We first show the state-of-the-art performance achieved by the proposed R-MVSNet on the recent MVS benchmarks. Then, we further demonstrate the scalability of the proposed method on several large-scale scenarios, where previous learned approaches often fail due to the memory constraint. Code is available at https://github.com/YoYo000/MVSNet.