G$^2$SR: Geometric Methods for Fast and Memory-Efficient Gaussian-based Surface Reconstruction
G2SR leverages multi-view geometry with lightweight neural detection to achieve fast (69-89 Hz), low-memory (203MB) Gaussian surface reconstruction, outperforming end-to-end methods.
Key Findings
Methodology
G2SR integrates a lightweight neural network for detecting 2D Gaussian splats and optical flow-based tracking across views. The core innovation lies in using classical multi-view geometry to analytically triangulate 3D splats from these correspondences. The neural frontend predicts splats in the reference view, while the backend employs an optimization based on the squared Hellinger distance to refine the 3D positions via Gauss-Newton iterations. This decouples the ill-posed detection/tracking subproblem from the well-posed triangulation, enabling high speed and generalization. The approach avoids large neural models, relying instead on geometric constraints for accurate, real-time surface reconstruction.
Key Results
- On datasets like ScanNet, Replica, and DTU, G2SR achieves geometric accuracy comparable or superior to state-of-the-art end-to-end methods, with depth errors of 4-8%. It operates at 69-89 reconstructions/sec on a single RTX 4090 GPU, with memory usage below 203MB, outperforming baselines by up to 18× in speed and 107× in memory efficiency.
- The method maintains metric-scale depth recovery without post-hoc alignment, demonstrating robustness across different camera configurations and scene complexities. Ablation studies confirm that the geometric backend significantly enhances accuracy and stability.
- Compared to neural-only approaches, G2SR's combination of geometric triangulation and lightweight detection reduces computational load, improves generalization, and enables real-time online applications in resource-constrained environments.
Significance
This work addresses the long-standing challenge of few-view scene reconstruction, especially on mobile and robotic platforms with limited resources. By integrating classical multi-view geometry with neural detection, G2SR offers a scalable, fast, and accurate solution that surpasses existing end-to-end methods in efficiency and robustness. Its low memory footprint and high throughput make it suitable for real-time applications such as autonomous exploration, AR, and industrial inspection, pushing the boundary of what is achievable in online scene understanding.
Technical Contribution
G2SR introduces a hybrid framework combining neural detection of 2D splats with an analytic triangulation backend based on multi-view geometry. It employs the squared Hellinger distance as an optimization metric, enabling precise, metric-scale 3D reconstruction without large neural models. The approach decouples the ill-posed detection/tracking from the well-posed triangulation, resulting in a lightweight, highly generalizable pipeline that operates efficiently on limited hardware. The method achieves a speed of up to 89 Hz with minimal memory, a significant leap over prior neural-only solutions.
Novelty
This is the first work to leverage classical multi-view geometry as the primary 3D solver within a neural detection pipeline for few-view Gaussian surface reconstruction. Unlike previous end-to-end neural methods that regress splats directly from images, G2SR exploits geometric constraints to analytically triangulate 3D splats, dramatically reducing model complexity and improving generalization. Its combination of neural detection with geometric triangulation represents a novel paradigm shift in efficient scene reconstruction.
Limitations
- The method's robustness may decline in scenes with extreme lighting variations, heavy occlusion, or highly reflective surfaces, which can impair detection and correspondence estimation.
- Performance drops in single-view scenarios, as the approach fundamentally relies on multi-view geometric constraints.
- Highly complex or geometrically ambiguous scenes might challenge the triangulation accuracy, necessitating further robustness enhancements.
Future Work
Future directions include integrating learned features to improve correspondence in challenging conditions, extending the framework to dynamic scenes, and scaling to larger environments. Combining this geometric approach with deep learning-based feature matching could further enhance robustness. Additionally, optimizing algorithms for even lower resource consumption and exploring real-time dynamic scene reconstruction are promising avenues.
AI Executive Summary
Reconstructing 3D scenes from limited views is a fundamental challenge in robotics and augmented reality. Existing neural approaches, while capable of high-fidelity results, demand large models and extensive computation, limiting their real-time applicability on resource-constrained devices. G2SR introduces a novel hybrid framework that marries classical multi-view geometry with lightweight neural detection, enabling fast, accurate, and memory-efficient surface reconstruction.
The core insight is that Gaussian splats, representing local scene patches, can be analytically triangulated from their multi-view projections once correspondences are established. G2SR employs a small neural network to detect 2D splats in a reference image, then uses optical flow to track these across views. The backend performs an optimization based on the squared Hellinger distance, refining the 3D positions through Gauss-Newton iterations. This division of labor allows the system to operate at 69-89 Hz with only 203MB of GPU memory, outperforming prior methods in both speed and resource usage.
Experimental results on datasets like ScanNet, Replica, and DTU demonstrate that G2SR achieves geometric accuracy comparable or superior to state-of-the-art end-to-end methods, with errors as low as 4-8%. Its ability to recover metric-scale depth without post-processing and maintain robustness across diverse scenes highlights its practical value. The approach's efficiency makes it ideal for real-time applications such as autonomous exploration, AR, and industrial inspection, especially on mobile platforms.
This work marks a significant step toward scalable, online Gaussian-based surface reconstruction, addressing key limitations of existing neural-only methods. By leveraging well-understood geometric principles, G2SR offers a promising pathway for future research in resource-efficient 3D scene understanding, with potential extensions into dynamic scenes and larger environments. Despite some limitations in extreme conditions, its innovative combination of neural detection and geometric triangulation sets a new standard for real-time, low-memory scene reconstruction.
Deep Dive
Abstract
Few-view surface reconstruction recovers the visible surfaces of a scene from a few posed RGB images, providing the 3D models that robots need to explore and interact online. On mobile platforms, the reconstruction must be fast and geometrically accurate while keeping a small memory footprint to ensure safe and efficient operation. 3D Gaussian Splatting (3DGS) offers a high-fidelity scene representation, but building it from a few views is ill-posed, as many distinct surfaces reproduce the same images, making traditional photometric methods prone to "floater" artifacts. End-to-end methods resolve the ambiguity by regressing splats with large, usually Transformer-based, networks that require heavy compute and memory while generalizing poorly to new scenes. We propose G2SR, which exploits a well-posed core of the task: given cross-view 2D splat correspondences, 3D splats follow analytically from multi-view geometry. G2SR employs a lightweight neural frontend to detect and track 2D Gaussian splats on the image plane and an analytic backend to triangulate each into a metric-scale 3D splat. On ScanNet, Replica, and DTU, G2SR matches or exceeds the geometric accuracy of state-of-the-art end-to-end methods while running at 69-89 reconstructions per second within 203 MB of GPU memory (5-107x less) for 2- and 3-view inputs at 384 x 512 resolution, offering a practical path to online Gaussian-based surface reconstruction.