PreF3R: Pose-Free Feed-Forward 3D Gaussian Splatting from Variable-length Image Sequence
PreF3R is a pose-free, feed-forward 3D Gaussian reconstruction method achieving 20FPS for real-time novel-view synthesis.
Key Findings
Methodology
PreF3R integrates a pretrained DUSt3R model with a spatial memory network to extend from pairwise to multi-view input, enabling continuous scene reconstruction without pose estimation. It employs a ViT encoder to extract features, a dense Gaussian prediction head to produce per-view pointmaps and Gaussian parameters, and differentiable rasterization for fast rendering. Training combines photometric and pointmap regression losses, ensuring high realism and structural accuracy. During inference, the model reconstructs the scene incrementally in a single feed-forward pass at 20FPS, supporting real-time applications.
Key Results
- On ScanNet++ and ARKitScenes, PREF3R surpasses MVSplat by 0.74dB and 1.95dB PSNR respectively at 10 views, achieving PSNR around 24.2dB, SSIM of 0.793, and LPIPS of 0.128, demonstrating superior quality.
- In 50-view scenarios, it maintains about 2.3 seconds per frame, outperforming traditional optimization-based methods in speed while preserving high fidelity.
- The model exhibits strong generalization, effectively reconstructing unseen scenes with high accuracy and realism, validating its robustness and scalability.
Significance
This work addresses the longstanding challenge of pose-free, real-time 3D scene reconstruction and novel-view synthesis. By removing the need for camera calibration and scene optimization, it simplifies the pipeline, making 3D understanding more accessible for AR, VR, robotics, and autonomous systems. Its ability to produce high-quality results rapidly opens new avenues for interactive and immersive applications, reducing computational costs and deployment complexity.
Technical Contribution
The key innovation is extending pretrained 2D-to-3D models with a spatial memory network, enabling continuous multi-view alignment without explicit pose estimation. The dense Gaussian prediction head allows end-to-end scene representation, while differentiable rasterization ensures fast, high-fidelity rendering. This architecture combines the strengths of deep learning and classical graphics, achieving a fully feed-forward, pose-free pipeline with real-time performance.
Novelty
PREF3R is the first to realize a pose-free, online, feed-forward 3D scene reconstruction and novel-view synthesis framework based solely on unposed image sequences. It leverages a spatial memory network to handle multiple views, avoiding scene-specific optimization, and predicts a 3D Gaussian field directly. This approach significantly advances the state-of-the-art by combining speed, accuracy, and generalization, filling a critical gap in the literature.
Limitations
- The method's accuracy depends on scale estimation; errors in scale can cause view drift and artifacts, especially in complex scenes.
- It currently performs best on indoor scenes with known scale, and struggles with outdoor or highly dynamic environments.
- While fast, the approach still faces challenges with extremely large scenes or very high-resolution inputs, which may require further optimization.
Future Work
Future research will focus on improving scale robustness, extending to dynamic scenes, and integrating self-supervised learning to reduce data dependency. Enhancing cross-scene generalization and scaling to larger, more complex environments are also key directions.
AI Executive Summary
Reconstructing 3D scenes and synthesizing novel views from unposed images has long been a core challenge in computer vision. Traditional methods rely heavily on camera calibration and scene-specific optimization, which are computationally intensive and limit real-time applications. Recent advances like NeRF and Gaussian Splatting have improved rendering quality but still depend on scene optimization, restricting their scalability and speed.
In this context, the authors introduce PreF3R, a novel framework that eliminates the need for camera poses and scene optimization. It leverages a pretrained 2D-to-3D model, DUSt3R, and extends it with a spatial memory network to handle sequential multi-view inputs. This architecture allows the model to predict a continuous 3D Gaussian field directly from unposed image sequences in a single feed-forward pass, achieving 20FPS on a single GPU. The core components include a ViT encoder for feature extraction, a dense Gaussian prediction head for scene representation, and differentiable rasterization for rendering.
Experimental results on large-scale datasets such as ScanNet++ and ARKitScenes demonstrate that PreF3R surpasses many existing pose-free methods in both quality and speed. It outperforms models like MVSplat and InstantSplat in PSNR, SSIM, and LPIPS metrics, especially in multi-view scenarios. Its ability to generalize to unseen scenes and operate in real-time marks a significant step forward, opening new possibilities for AR, VR, robotics, and autonomous navigation.
Despite these advances, challenges remain, including sensitivity to scale estimation and performance in outdoor or dynamic environments. Future work aims to address these issues by enhancing robustness, extending to dynamic scenes, and exploring self-supervised training strategies. Overall, PreF3R represents a major breakthrough in pose-free, real-time 3D scene understanding, promising to accelerate the deployment of intelligent spatial perception systems across industries.
Deep Dive
Key Concepts
Gaussian Splatting (高斯点云)
一种用高斯分布表示3D点的技术，支持高效、可微的渲染，适合场景重建与新视角生成。
空间记忆网络
一种存储和融合多视角信息的机制，帮助模型连续对齐多帧图像，实现无姿态场景重建。
差分光栅化
一种可微的图像投影技术，将高斯场快速渲染到不同视角，支持端到端训练。
端到端前馈模型
无需场景优化，直接从输入图像预测3D场景和新视角图像的深度学习架构。
多视角连续重建
在未配准的多视角图像序列中，连续构建场景的3D模型，实现实时交互。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在极端场景（如户外、动态环境）中的鲁棒性，仍是未来研究的关键。当前模型对尺度估计敏感，误差会影响渲染质量，需开发更稳健的尺度估计机制。
Abstract
We present PreF3R, Pose-Free Feed-forward 3D Reconstruction from an image sequence of variable length. Unlike previous approaches, PreF3R removes the need for camera calibration and reconstructs the 3D Gaussian field within a canonical coordinate frame directly from a sequence of unposed images, enabling efficient novel-view rendering. We leverage DUSt3R's ability for pair-wise 3D structure reconstruction, and extend it to sequential multi-view input via a spatial memory network, eliminating the need for optimization-based global alignment. Additionally, PreF3R incorporates a dense Gaussian parameter prediction head, which enables subsequent novel-view synthesis with differentiable rasterization. This allows supervising our model with the combination of photometric loss and pointmap regression loss, enhancing both photorealism and structural accuracy. Given a sequence of ordered images, PreF3R incrementally reconstructs the 3D Gaussian field at 20 FPS, therefore enabling real-time novel-view rendering. Empirical experiments demonstrate that PreF3R is an effective solution for the challenging task of pose-free feed-forward novel-view synthesis, while also exhibiting robust generalization to unseen scenes.