Off The Grid: Detection of Primitives for Feed-Forward 3D Gaussian Splatting

The evolution of neural scene reconstruction has transitioned from volumetric radiance fields like NeRF to point-based and Gaussian primitive models, aiming for faster rendering and better scalability. Early methods relied on explicit camera poses and scene geometry, which limited real-time applications. Recent advances introduced pose-free models and self-supervised learning, reducing dependency on annotations. Techniques like PixelSplat and voxel-aligned Gaussian models improved efficiency but suffered from rigid grid constraints, leading to artifacts and limited detail capture. The integration of large-scale pre-trained models like VGGT further enhanced geometry estimation, yet primitive placement strategies remained largely grid-dependent, constraining scene fidelity. Addressing these limitations, recent research emphasizes flexible, learnable primitive distributions, but a comprehensive solution for sub-pixel, adaptive placement without pose supervision was lacking.

Current pose-free neural scene reconstruction methods predominantly utilize pixel- or voxel-aligned primitives, which restrict the expressiveness and detail fidelity of the reconstructed scenes. These approaches often produce blurry or artifact-prone outputs, especially in complex or sparse-view scenarios. The core challenge lies in designing a primitive placement strategy that is both flexible and precise, capable of capturing high-frequency details without excessive primitive count or computational burden. Additionally, existing methods lack effective mechanisms for adaptive distribution of primitives based on scene content, leading to inefficient resource allocation. Overcoming these bottlenecks is crucial for achieving real-time, high-quality scene synthesis in diverse environments.

The paper introduces a novel 'Off-The-Grid' detection framework that employs a sub-pixel Gaussian detection decoder inspired by keypoint detection techniques. This allows primitives to be placed continuously in image space, avoiding the rigidity of grid-based layouts. A multi-density adaptive mechanism dynamically allocates primitives based on local scene complexity, measured via Shannon entropy, ensuring detailed regions receive more primitives. The integration with a large-scale pre-trained 3D reconstruction backbone (VGGT) enables the model to learn primitive placement directly from images in an end-to-end, pose-free manner. This approach significantly reduces primitive redundancy, enhances scene detail, and improves rendering fidelity, marking a departure from traditional pixel- or voxel-aligned methods.

• �� Use VGGT to extract initial scene geometry from unposed images, generating depth and camera parameters.
• �� Design a U-Net-based decoder that takes concatenated image features, depth maps, and tokens from VGGT, producing heatmaps for primitive detection.
• �� Apply a softmax-based heatmap to obtain continuous 2D Gaussian centers via a differentiable soft-argmax operation.
• �� Implement a multi-density mechanism: compute Shannon entropy for each image patch, assign primitives into low, medium, and high-density groups accordingly.
• �� Unproject 2D centers into 3D space using interpolated depth and camera intrinsics, placing Gaussian centers in scene coordinates.
• �� Predict primitive parameters—scale, orientation (quaternions), opacity, confidence—via small MLPs from interpolated descriptors.
• �� Transform primitives from camera to world coordinates using predicted extrinsics, fuse multi-view primitives weighted by confidence and opacity.
• �� Train end-to-end with photometric losses (L1, SSIM, LPIPS) and geometric consistency losses (depth, normals), regularized by a teacher VGGT model to stabilize training.
• �� During inference, prune primitives with low confidence and opacity, rendering the scene with a custom rasterizer for high fidelity.

The model was trained on diverse datasets including DL3DV, Charge, and Tanks and Temples, with input views ranging from 2 to 12. Evaluation metrics included PSNR, SSIM, and LPIPS, comparing against baselines like PixelSplat and voxel-based Gaussian models. Ablation studies assessed the impact of the sub-pixel detection and multi-density mechanisms. Hyperparameters such as primitive density levels, entropy thresholds, and learning rates were tuned for optimal performance. The experiments demonstrated that the proposed method achieved superior quantitative scores and visual quality, especially in complex scenes and extrapolated views. The robustness was validated through cross-dataset testing, confirming generalization and scene fidelity improvements.

Quantitative results show PSNR of 21.21, SSIM of 0.647, and LPIPS of 0.353 at 12 views, outperforming prior methods by significant margins. The primitive count was only 14.3% of input pixels, yet scene details were sharper, with fewer artifacts. Ablation confirmed that sub-pixel detection and adaptive density contributed over 70% performance gains. Visualizations revealed crisper geometries and textures, especially under extreme viewpoints, with the model effectively balancing detail and computational efficiency. These results demonstrate the method’s ability to produce high-fidelity, real-time scene reconstructions with minimal primitives.

This approach is ideal for real-time scene synthesis in virtual reality, AR/VR applications, and digital content creation, where high detail and efficiency are critical. It requires only multi-view images, without camera pose annotations, making it accessible for consumer devices and large-scale scene capture. The method can also facilitate scene understanding in robotics and autonomous navigation, providing accurate, detailed 3D models rapidly. Future integration with dynamic scene modeling could further expand its industry impact.

Dependence on pre-trained VGGT geometry limits performance in scenes with complex, non-rigid, or highly occluded objects. High-density primitive allocation may incur computational costs, restricting scalability. The current static scene focus restricts real-time dynamic scene reconstruction. Extending to non-rigid scenes and optimizing for large-scale environments remain open challenges, requiring further research into adaptive, scalable architectures.