Gaussian Sculpting: End-to-End Controllable Surface Reconstruction via Field Optimization

The evolution of 3D reconstruction has transitioned from traditional multi-view stereo (MVS) and volumetric methods to neural implicit representations. Early techniques like Poisson surface reconstruction and MVS relied on dense point clouds and mesh generation, but struggled with sparse views and noise. Neural approaches such as NeRF introduced continuous volumetric radiance fields, enabling photo-realistic novel view synthesis but lacked explicit surface control. Surface reconstruction networks like DeepSDF and Occupancy Networks provided explicit geometry but faced challenges in detail preservation and view limitations. Recently, Gaussian Splatting (3DGS) achieved real-time rendering with Gaussian primitives but lacked integrated surface refinement, leading to geometric inaccuracies. Current research aims to combine the efficiency of Gaussian primitives with precise surface optimization, addressing the limitations of prior methods in sparse-view scenarios and complex geometries.

The core challenge is achieving accurate, high-fidelity surface reconstruction from limited viewpoints using neural representations. Existing methods either rely on multi-stage pipelines, which are complex and less flexible, or lack explicit surface control, leading to geometric errors and missing details. Gaussian primitives, while efficient for rendering, are irregular and difficult to optimize directly for surface accuracy. The key bottleneck is integrating primitive-based representations with differentiable surface refinement, ensuring stability and detail preservation under view constraints. Addressing these issues requires a unified framework that can optimize geometry and appearance simultaneously, with mechanisms to enforce consistency and preserve fine details.

This work introduces a unified framework that anchors Gaussian primitives onto an evolving differentiable surface, enabling direct, end-to-end optimization of geometry and appearance. The main innovations include: 1) a bi-level training strategy separating geometry and parameter updates for stable gradient flow; 2) parameter constraints that enforce Gaussian-surface consistency; 3) an octree-like multi-resolution subdivision scheme that adaptively refines surface details while controlling memory usage; 4) integration of these components into a fully differentiable pipeline, allowing joint optimization of signed distance fields and Gaussian parameters. These innovations collectively improve surface accuracy, detail preservation, and robustness in view-limited scenarios, surpassing prior methods like 3DGS and neural voxel-based approaches.

• �� Initialize Gaussian primitives and anchor them onto a differentiable surface. • Construct a signed distance field (SDF) representing the geometry. • Implement a bi-level training loop: • - Outer loop updates the SDF geometry via gradient descent based on reconstruction loss. • - Inner loop fixes the geometry and updates Gaussian parameters (mean, covariance, amplitude) to fit the current surface. • Impose constraints on Gaussian parameters to ensure they align with the surface geometry. • Use octree-like subdivision to adaptively refine the surface at multiple resolutions, balancing detail and memory. • During training, propagate gradients through the entire pipeline, updating both the SDF and Gaussian parameters jointly. • Regularize parameters to prevent drift and ensure consistency. • Finalize the surface by extracting the optimized geometry, which accurately captures the scene’s details.

The authors evaluate their method on object-level datasets, comparing against 3DGS, Neural Sparse Voxel, and DeepSDF. Metrics include PSNR, Chamfer Distance, and SSIM. Training uses Adam optimizer with a learning rate of 0.001 over 200 epochs. Ablation studies assess the impact of bi-level training, parameter constraints, and multi-resolution subdivision. Tests include varying view counts and resolutions to evaluate robustness. Results demonstrate superior geometric accuracy, detail preservation, and ability to recover missing structures under limited viewpoints. Visualizations highlight the removal of artifacts and improved surface smoothness, confirming the effectiveness of the proposed components.

Quantitative results show a PSNR of 32.5dB, outperforming baseline methods by approximately 3dB, with a 20% reduction in Chamfer Distance. The multi-resolution subdivision preserves fine details at low resolutions, with ablation studies indicating a 15% drop in accuracy when components are removed. The method effectively reconstructs complex geometries, recovering missing parts caused by viewpoint limitations, and reduces artifacts such as surface noise and redundant surfaces. These results validate the approach’s ability to produce high-quality, controllable surfaces suitable for downstream applications.

This framework is applicable in virtual reality content creation, cultural heritage digitization, industrial inspection, and AR/VR asset generation. It requires multi-view images or point clouds as input and can produce detailed surface meshes suitable for rendering, analysis, or editing. Its end-to-end nature simplifies integration into existing pipelines, enabling scalable high-fidelity reconstructions. Future applications may include dynamic scene reconstruction, real-time editing, and semantic-aware modeling, broadening the scope of neural surface methods in industry and academia.

Despite its strengths, the method faces challenges in scaling to very large scenes due to memory and computational demands of multi-scale subdivision. Handling dynamic scenes with temporal coherence remains unresolved. The parameter constraints, while stabilizing, may limit the representation of highly intricate geometries, leading to potential oversmoothing. Moreover, the approach's reliance on high-quality multi-view data limits its robustness in noisy or sparse input scenarios. Future work should focus on optimizing computational efficiency, extending to dynamic environments, and incorporating semantic priors for improved generalization.