ZipSplat: Fewer Gaussians, Better Splats
ZipSplat uses scene tokens to decouple Gaussian placement from pixels, achieving high-quality scene reconstruction with 6× fewer Gaussians, and adjustable quality-efficiency trade-off.
Key Findings
Methodology
ZipSplat employs a multi-view foundation model to extract dense visual tokens, which are compressed via k-means clustering into scene tokens. These tokens are refined through cross- and self-attention layers, then decoded by a lightweight MLP into groups of Gaussians with unconstrained 3D positions. During inference, the clustering ratio r is adjustable, enabling a continuous quality-efficiency trade-off without retraining. The model is trained with geometric supervision (Chamfer loss) and progressive schedules to ensure Gaussians align with scene surfaces. This approach removes the pixel-grid coupling, allowing the model to allocate Gaussians adaptively based on scene complexity, significantly reducing the total number of Gaussians needed while maintaining or improving rendering quality.
Key Results
- On DL3DV, ZipSplat with 15K Gaussians matches or exceeds the quality of pixel-aligned methods using 380K Gaussians, with a 2.1dB PSNR gain. It maintains stable performance as view count increases, outperforming baselines like YoNoSplat and C3G. Cross-dataset tests on Mip-NeRF360 and ScanNet++ show zero-shot generalization surpassing all comparable methods. Adjusting the clustering ratio during inference provides a flexible quality-efficiency trade-off, with performance gains of about 5dB PSNR after test-time optimization.
- The model achieves state-of-the-art pose-free view synthesis results, predicting 6× fewer Gaussians than pixel-aligned counterparts, with PSNR improvements of 2.1dB on DL3DV and 1.2dB on RealEstate10K. It demonstrates robustness across varying scene complexities and view counts, with qualitative results showing sharper geometry and fewer artifacts. The ability to fine-tune Gaussian budgets at inference offers practical benefits for real-time applications.
Significance
This work fundamentally advances scene representation by decoupling Gaussian placement from pixel grids, enabling more compact, flexible, and scalable 3D scene reconstructions. It addresses longstanding limitations of pixel-aligned methods, such as redundancy and inefficiency, especially in unobserved or occluded regions. The approach opens new avenues for real-time, large-scale scene modeling in AR/VR, digital twins, and visual effects industries, reducing hardware and data requirements while improving quality. Its zero-shot generalization capability further broadens its impact across diverse datasets and applications.
Technical Contribution
ZipSplat introduces a novel scene token-based architecture that replaces pixel-aligned Gaussian prediction with a dynamic, scene-level representation. The core innovations include feature-space k-means clustering for adaptive Gaussian budget control, a lightweight MLP for free 3D placement, and attention mechanisms for token refinement. The model’s ability to adjust the Gaussian count at inference time without retraining is a significant technical breakthrough, enabling flexible quality-efficiency trade-offs. Geometric supervision ensures the meaningful placement of Gaussians, while the entire pipeline maintains high fidelity with fewer primitives.
Novelty
This is the first work to decouple Gaussian placement from the pixel grid via scene tokens, enabling adaptive, scene-aware primitive distribution. Unlike prior pixel-aligned approaches, ZipSplat predicts Gaussians directly in 3D space, guided by learned tokens, which allows for significant reduction in primitive count without sacrificing detail. Its ability to adjust the Gaussian budget dynamically during inference sets it apart, providing a new paradigm for scalable, pose-free scene reconstruction.
Limitations
- The model’s reliance on geometric supervision may limit performance in scenes with sparse or noisy depth cues. Extreme scene complexity or highly unstructured environments could challenge the Gaussian placement accuracy.
- While the inference-time adjustment of compression ratio offers flexibility, very high compression ratios can lead to detail loss, especially in fine structures.
- The approach’s effectiveness in dynamic scenes or large-scale outdoor environments remains to be validated, and computational costs for very high-resolution outputs could be significant.
Future Work
Future directions include integrating dynamic scene modeling, improving geometric supervision in unstructured environments, and exploring more sophisticated token generation strategies. Extending the approach to handle large-scale outdoor scenes and real-time dynamic updates will be key for industrial applications. Additionally, combining this framework with learned priors or multi-modal data could further enhance scene understanding and reconstruction fidelity.
AI Executive Summary
Reconstructing detailed 3D scenes from sparse images has long been a challenge in computer vision, with traditional methods like neural radiance fields (NeRF) offering high fidelity but at prohibitive computational costs. Recent advances introduced explicit primitives such as 3D Gaussian splatting, which accelerate rendering but rely heavily on pixel-aligned predictions. These pixel-based methods tie the number of primitives directly to camera resolution, leading to inefficiencies and redundant primitives in overlapping views.
Veicht et al. propose ZipSplat, a novel scene representation framework that fundamentally decouples Gaussian placement from the pixel grid by leveraging scene tokens. This approach extracts dense visual tokens from multiple views using a foundation model, compresses them via feature-space k-means clustering, and refines them through attention mechanisms. Each token is then decoded into a set of Gaussians with unconstrained 3D positions, allowing the model to adaptively allocate primitives based on scene complexity. During inference, the clustering ratio can be tuned to balance quality and efficiency, enabling a single trained model to operate across a spectrum of reconstruction fidelities.
The core technical innovation lies in replacing the pixel-aligned primitive prediction with a token-based, scene-aware primitive distribution. This not only reduces the total number of Gaussians needed—by up to six times—but also improves detail preservation, especially in complex regions. Geometric supervision ensures that Gaussians align with scene surfaces, while progressive training schedules stabilize learning. Extensive experiments on datasets like DL3DV and RealEstate10K demonstrate that ZipSplat surpasses state-of-the-art pose-free methods, achieving higher PSNR and SSIM scores with fewer primitives. Its zero-shot generalization to datasets like Mip-NeRF360 and ScanNet++ further underscores its robustness.
This work marks a significant step toward scalable, real-time scene reconstruction without the need for precise camera poses or dense input data. By enabling flexible quality-efficiency trade-offs at inference, ZipSplat opens new avenues for applications in AR/VR, digital twins, and large-scale environment modeling. Despite some limitations in highly unstructured or dynamic scenes, its innovative token-based paradigm paves the way for future research in efficient 3D scene understanding and rendering.
Deep Dive
Abstract
Feed-forward 3D Gaussian Splatting methods reconstruct a scene from posed or pose-free images in a single forward pass, yet current approaches predict one Gaussian per input pixel, tying the representation budget to camera resolution rather than scene complexity. A flat wall and a richly textured object thus produce equally many Gaussians despite very different geometric needs. We propose ZipSplat, a token-based feed-forward model that decouples Gaussian placement from the pixel grid. A multi-view backbone extracts dense visual tokens, and k-means clustering compresses them into a compact set of scene tokens. Cross- and self-attention refine these tokens, and a lightweight MLP decodes each into a group of Gaussians with unconstrained 3D positions. Because clustering is applied at inference, a single trained model spans the quality-efficiency curve without retraining. ZipSplat operates without ground-truth poses or intrinsics, yet sets a new state of the art on DL3DV and RealEstate10K with ${\sim}6{\times}$ fewer Gaussians than pixel-aligned methods, surpassing the best pose-free baseline by 2.1dB and 1.2dB PSNR, respectively. It further generalizes zero-shot to Mip-NeRF360 and ScanNet++, outperforming all comparable baselines. Our project page is at https://veichta.com/zipsplat.