Gaussian On-the-Fly Splatting: A Progressive Framework for Robust Near Real-Time 3DGS Optimization
Proposes On-the-Fly GS, a progressive framework enabling near real-time 3DGS optimization, reducing per-image optimization to seconds.
Key Findings
Methodology
Introduces On-the-Fly GS, a progressive framework combining On-the-Fly SfM and local & semi-global optimization strategies to dynamically update the 3DGS field. Adaptive learning rate scheduling and load balancing enhance training stability and efficiency.
Key Results
- Optimization time per image reduced to seconds across benchmarks, achieving comparable rendering quality to offline methods (PSNR improved to 31.93 dB).
- Rendering quality significantly improved over GS-SLAM (SSIM increased from 0.58 to 0.86) while supporting discrete image inputs.
- Model size reduced by approximately 50% on average, with training efficiency boosted (FPS increased to 0.36).
Significance
Addresses the limitations of offline 3DGS training by enabling real-time scene expansion, accelerating post-capture optimization, and advancing dynamic scene reconstruction technologies.
Technical Contribution
Proposes local & semi-global optimization strategies, adaptive learning rate scheduling, and load balancing, significantly improving training efficiency. Supports discrete image inputs, broadening 3DGS applicability.
Novelty
First to achieve near real-time 3DGS optimization for dynamically expanding scenes, overcoming SLAM-based methods' reliance on spatiotemporal continuity and depth sensors.
Limitations
- Requires sufficient overlap between images; low-overlap scenarios may degrade optimization performance.
- Currently supports single-GPU execution, limiting multi-GPU parallelization.
- Training efficiency for large-scale scenes still requires improvement.
Future Work
Future directions include exploring multi-GPU parallelization, strategies for low-overlap image optimization, and improving efficiency for large-scale scene training.
AI Executive Summary
Traditional 3DGS methods rely on offline training, making them unsuitable for real-time scene expansion, which limits their applicability in dynamic scenarios. To address this, the authors propose On-the-Fly GS, a progressive framework enabling near real-time 3DGS optimization during image capture.
The framework dynamically updates image poses and sparse point clouds during acquisition and employs local & semi-global optimization strategies to prioritize training for new images and their neighbors. Adaptive learning rate scheduling and load balancing further stabilize the training process, significantly improving efficiency and rendering quality.
Experiments demonstrate that On-the-Fly GS achieves outstanding performance across benchmarks, reducing per-image optimization time to seconds while maintaining rendering quality comparable to offline methods. This research offers a novel solution for dynamic scene reconstruction and expands the applicability of 3DGS technology.
Deep Dive
Abstract
3D Gaussian Splatting (3DGS) achieves high-fidelity rendering with fast real-time performance, but existing methods rely on offline training after full Structure-from-Motion (SfM) processing. In contrast, this work introduces Gaussian on-the-fly Splatting (abbreviated as On-the-Fly GS), a progressive framework enabling near real-time 3DGS optimization during image capture. As each image arrives, its pose and sparse points are updated via On-the-Fly SfM, and newly optimized Gaussians are immediately integrated into the 3DGS field. To achieve this, we propose a progressive Local & Semi-Global optimization to prioritize the new image and its neighbors by their corresponding overlapping relationship, allowing the new image and its overlapping images to get more training. To further stabilize training across previous and new images, an adaptive learning rate schedule balances the iterations and the learning rate. Extensive experiments on multiple benchmarks show that our On-the-Fly GS reduces training time significantly, optimizing each new image in seconds with minimal rendering loss, offering one of the first practical steps toward rapid, progressive 3DGS reconstruction.