Scaling 3D Generative Priors to Large-Scale Scene Meshes from Multi-View Images
A training-free large-scale scene mesh generation framework combining local continuity and global appearance consistency, improving geometric and visual fidelity.
Key Findings
Methodology
This paper builds upon the pretrained 3D generative model TRELLIS2, proposing a training-free framework for multi-view scene mesh synthesis. The approach includes adaptive scene decomposition based on point cloud density, local context tiled generation with neighbor-aware attention, and global appearance alignment via cross-voxel feature matching. The system employs sparse voxel encoding and multi-view feature fusion to maintain high detail. Each component addresses specific challenges: scene partitioning ensures resolution consistency; neighbor attention reduces boundary discontinuities; global alignment minimizes distant appearance discrepancies. The pipeline integrates these modules into a coherent, scalable process capable of reconstructing detailed large-scale scenes without additional training.
Key Results
- On UrbanScene3D, the method achieved LPIPS of 0.241, outperforming Extend3D's 0.412, indicating superior visual fidelity. Chamfer distance was reduced to 0.015, and F-score reached 0.925, demonstrating high geometric accuracy. Ablation studies confirmed that each component contributed to improvements in continuity and appearance. Increasing tile count from 3 to 7 further refined details, with LPIPS decreasing from 0.683 to 0.605. The approach effectively leverages multi-view data, producing more realistic urban scene reconstructions compared to single-view baselines.
- Quantitative results show that the adaptive scene decomposition aligns tile resolution with scene complexity, enabling finer details. Neighbor-aware attention enhances boundary smoothness, while global feature matching ensures consistent appearance across distant regions. These combined strategies lead to reconstructions with fewer artifacts, higher fidelity, and better structural coherence, validated through extensive experiments across diverse urban and synthetic scenes.
Significance
This work advances large-scale 3D scene reconstruction by enabling high-fidelity, scalable, and training-free mesh generation from multi-view images. It addresses longstanding issues of boundary discontinuity and appearance inconsistency in tiled generation, making it highly relevant for applications in virtual reality, urban modeling, and digital twins. By integrating pre-trained models with adaptive scene decomposition and cross-region feature alignment, it reduces the need for scene-specific training data, broadening practical deployment. The framework's ability to produce detailed, coherent large-scale scenes paves the way for more immersive, realistic virtual environments, and offers a scalable solution for industry and research needs.
Technical Contribution
The key technical innovations include an adaptive scene decomposition strategy that dynamically determines tile count based on scene geometry, neighbor-aware self-attention modules that incorporate boundary context during tile decoding, and a global appearance alignment mechanism utilizing cross-voxel feature matching. These components collectively enable high-resolution, continuous, and visually consistent large-scale scene reconstructions without retraining the pretrained 3D generator. The approach extends the capabilities of existing models like TRELLIS2, overcoming their resolution limitations and scalability issues, and introduces a novel multi-scale, multi-view collaborative framework that maintains geometric and appearance fidelity across extensive environments.
Novelty
This study is the first to realize training-free, multi-view large-scale scene mesh generation by combining adaptive scene decomposition, neighbor-aware attention, and global feature alignment. Unlike prior methods limited to object-centric or scene-specific training, it leverages pretrained models for scalable outdoor scene reconstruction. Its core innovation lies in integrating local and global consistency mechanisms within a tiled generation framework, enabling detailed, seamless, and appearance-coherent large environments without additional training or fine-tuning.
Limitations
- 依赖于稠密点云重建质量，点云稀疏或噪声较大时，几何细节可能不足。多视图匹配在遮挡或纹理不足场景中表现有限，影响整体效果。
- 计算资源消耗较大，尤其在高切片数和复杂场景中，内存和时间成本较高，需优化算法效率。
- 尚未支持动态场景或实时交互，未来需结合实时感知技术实现动态更新。
Future Work
未来将探索端到端训练策略，结合深度学习优化场景分解和特征匹配，提升效率与效果。还计划扩展到动态场景和实时应用，结合传感器数据实现实时场景更新，推动虚拟现实、自动驾驶等行业的应用落地。
AI Executive Summary
This work introduces a novel, training-free framework for large-scale 3D scene mesh generation from multi-view images, leveraging pretrained models like TRELLIS2. The core challenge addressed is maintaining geometric continuity and appearance consistency across extensive environments, which traditional tiled approaches struggle with. The proposed solution employs adaptive scene decomposition based on point cloud density, ensuring each scene segment is within the native resolution of the pretrained generator. To enhance boundary smoothness, neighbor-aware self-attention modules incorporate local context during tile decoding, reducing boundary artifacts. Additionally, a global appearance alignment mechanism matches features across distant scene regions, ensuring consistent textures and colors throughout the scene.
Experimental results on UrbanScene3D demonstrate the effectiveness of this approach. Quantitative metrics show LPIPS scores of 0.241, outperforming existing methods such as Extend3D and GenRecon. Geometric accuracy, measured by Chamfer distance, improved to 0.015, and F-score reached 0.925, indicating high fidelity in both shape and appearance. The ablation studies confirm that each component—adaptive decomposition, neighbor attention, and global alignment—contributes significantly to the final quality. The framework scales well with increased tile resolution, enabling finer details and more realistic urban reconstructions.
This research pushes the boundary of large-scale scene modeling, providing a scalable, high-fidelity, and training-free solution suitable for applications in virtual reality, urban planning, and digital twins. It addresses critical issues of boundary discontinuity and appearance inconsistency, offering a practical pathway toward more immersive and accurate virtual environments. Despite current limitations in computational cost and scene dynamics, the proposed framework sets a foundation for future enhancements in real-time, dynamic scene reconstruction, and broader deployment across diverse domains.
Deep Dive
Abstract
Pretrained 3D generative models produce detailed geometry and appearance but are primarily designed for object-centric generation within a limited spatial extent. Recent approaches address this limitation by partitioning large scenes into smaller spatial regions and applying pretrained 3D generative priors to each region. However, scaling tiled generation to large multi-view scenes makes it challenging to maintain local geometric continuity and global appearance consistency. We present a training-free framework for large-scale textured mesh generation from multi-view images. Our key idea is to scale tiled generation to large scenes with increased spatial detail while coordinating generation both locally and globally. We introduce local context tiled generation to improve geometric continuity between neighboring regions and global appearance alignment to reduce appearance discrepancies across distant regions. An adaptive scene decomposition further determines the number of tiles according to the input scene geometry. Experiments demonstrate improved geometric and appearance fidelity over existing approaches while enabling fine-grained generation of large-scale scenes.