Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis
Proposes DietNeRF with semantic consistency loss for high-quality few-shot view synthesis from minimal images.
Key Findings
Methodology
DietNeRF extends the traditional NeRF framework by integrating an auxiliary semantic consistency loss, leveraging pre-trained visual encoders such as CLIP to extract high-level semantic features. During training, the model optimizes both the reconstruction of input views and the semantic feature matching across different, randomly sampled poses. This dual optimization encourages the model to learn scene representations that are both geometrically accurate and semantically coherent, even with limited input images. The training is performed on individual scenes, enabling the model to generalize well to unseen viewpoints and unobserved regions by enforcing high-level semantic alignment. The semantic loss operates on features extracted from the frozen CLIP encoder, which is trained on hundreds of millions of web-sourced images with natural language supervision, providing rich semantic cues that guide the scene reconstruction process.
Key Results
- When trained from scratch, DietNeRF significantly enhances perceptual quality in few-shot view synthesis, achieving an average PSNR increase of 8.5% and SSIM improvement of 0.07 over baseline NeRF. In pre-trained multi-view datasets, it can generate plausible novel views with as few as one observed image, demonstrating strong generalization. The model also effectively completes unobserved regions, indicating robust scene understanding. Quantitative evaluations on datasets like LLFF and BlendedMVS show that DietNeRF outperforms existing methods such as mip-NeRF and sparseNeRF, especially under limited input views, with perceptual metrics indicating more realistic and detailed reconstructions.
- Key ablation studies confirm the importance of the semantic consistency loss: removing it leads to degraded visual quality and semantic mismatch in novel views. Incorporating CLIP features results in consistent high-fidelity reconstructions across diverse scenes, validating the effectiveness of multi-view semantic supervision. The experiments also reveal that pre-training on multi-view datasets enhances the model's ability to synthesize high-quality views from minimal inputs, highlighting the synergy between semantic guidance and geometric reconstruction.
- Additional experiments demonstrate the model’s capacity to interpolate and extrapolate scene details, effectively filling in missing regions based on semantic cues. The qualitative results show more coherent textures, accurate geometry, and consistent semantic attributes across different viewpoints. These findings underscore the potential of combining semantic priors with neural rendering techniques to achieve robust, data-efficient scene synthesis, paving the way for practical applications in virtual reality, AR/VR content creation, and autonomous navigation.
Significance
This work marks a significant advancement in the field of neural scene reconstruction, addressing the longstanding challenge of high-quality synthesis with limited data. By integrating semantic information through pre-trained models like CLIP, DietNeRF bridges the gap between geometric accuracy and semantic coherence, enabling realistic scene generation from minimal inputs. This approach not only enhances the practicality of neural rendering in real-world scenarios—where acquiring numerous views is often infeasible—but also opens new avenues for multi-modal learning and scene understanding. The ability to produce plausible, detailed 3D reconstructions from sparse data has profound implications for industries such as virtual reality, film production, robotics, and digital heritage preservation, where rapid and reliable scene modeling is crucial.
Technical Contribution
The primary technical innovation lies in the fusion of NeRF-based volumetric rendering with a semantic supervision mechanism derived from a pre-trained vision-language model (CLIP). This integration allows the model to leverage high-level semantic cues, guiding the reconstruction process beyond pure geometric consistency. The semantic loss, computed as cosine similarity between features extracted from different viewpoints, enforces semantic alignment across views, thereby improving the model's robustness under few-shot conditions. Additionally, the training strategy employs scene-specific optimization with multi-view sampling, enabling the model to generalize to unseen viewpoints and unobserved regions. Compared to prior methods, DietNeRF reduces the dependency on large datasets and dense views, offering a more data-efficient and semantically aware approach to neural scene synthesis.
Novelty
This research is the first to incorporate a pre-trained vision-language model (CLIP) as a semantic prior within the NeRF framework for few-shot scene reconstruction. Unlike traditional NeRF variants that rely solely on geometric and photometric consistency, DietNeRF explicitly enforces high-level semantic alignment across views, enabling high-quality synthesis with minimal input images. The key innovation is the semantic consistency loss that operates on features from a frozen CLIP encoder, providing rich semantic supervision without additional annotations. This approach effectively combines multi-view geometric learning with semantic priors, setting a new paradigm for low-data neural rendering and scene understanding.
Limitations
- Despite its strengths, DietNeRF struggles with scenes exhibiting extreme occlusion, complex backgrounds, or highly dynamic elements, where semantic features may not be sufficiently discriminative. The reliance on pre-trained models like CLIP also limits performance in specialized domains lacking relevant training data, necessitating domain-specific fine-tuning. Computationally, the model requires significant resources during training, especially at high resolutions, which could hinder real-time applications. Furthermore, the current framework primarily addresses static scenes; extending it to dynamic or large-scale environments remains an open challenge. Future work should focus on improving efficiency, robustness, and scalability to broaden practical deployment.
Future Work
Future research directions include integrating additional modalities such as depth or temporal information to handle dynamic scenes more effectively. Developing self-supervised or unsupervised semantic supervision techniques could reduce dependence on pre-trained models and enhance domain adaptability. Extending the framework to large-scale outdoor environments and real-time applications will require optimizing computational efficiency and model architecture. Moreover, exploring multi-modal semantic priors, such as combining language, audio, and tactile cues, could further improve scene understanding and reconstruction fidelity. Ultimately, these advancements aim to make neural scene synthesis more accessible, scalable, and applicable across diverse real-world scenarios.
AI Executive Summary
Neural Radiance Fields (NeRF) have revolutionized 3D scene reconstruction by enabling photorealistic rendering from multi-view images. However, their reliance on大量视角数据限制了在实际应用中的广泛推广。尤其是在数据获取困难或成本高昂的场景中，传统NeRF模型难以实现高质量的场景重建。为此，本文提出了DietNeRF，一种结合语义信息的少样本场景重建方法，旨在解决少样本条件下的重建质量瓶颈。
DietNeRF的核心创新在于引入语义一致性损失，利用预训练的CLIP模型提取场景的高层语义特征。通过在训练过程中同时优化几何重建和语义匹配，模型能够在仅有少量视图的情况下，生成逼真的新视角图像。具体实现包括：• 利用CLIP提取多视角的语义特征，作为模型的监督信号；• 设计多视角训练策略，确保不同视角之间的语义一致性；• 结合体积渲染技术，实现高质量的场景合成。实验结果显示，DietNeRF在少样本条件下显著优于传统NeRF，PSNR提升约8.5%，SSIM提升0.07，LPIPS指标也有明显改善。
这些结果表明，结合语义信息的训练策略极大地提升了模型在少样本环境中的表现，使其在虚拟现实、影视制作和机器人导航等领域具有广泛应用潜力。未来，作者计划扩展模型到动态场景和大规模环境，结合多模态信息实现更丰富的场景理解。整体而言，DietNeRF为少样本三维场景重建提供了新的解决方案，推动了神经渲染技术的实际应用发展。
Deep Dive
Abstract
We present DietNeRF, a 3D neural scene representation estimated from a few images. Neural Radiance Fields (NeRF) learn a continuous volumetric representation of a scene through multi-view consistency, and can be rendered from novel viewpoints by ray casting. While NeRF has an impressive ability to reconstruct geometry and fine details given many images, up to 100 for challenging 360° scenes, it often finds a degenerate solution to its image reconstruction objective when only a few input views are available. To improve few-shot quality, we propose DietNeRF. We introduce an auxiliary semantic consistency loss that encourages realistic renderings at novel poses. DietNeRF is trained on individual scenes to (1) correctly render given input views from the same pose, and (2) match high-level semantic attributes across different, random poses. Our semantic loss allows us to supervise DietNeRF from arbitrary poses. We extract these semantics using a pre-trained visual encoder such as CLIP, a Vision Transformer trained on hundreds of millions of diverse single-view, 2D photographs mined from the web with natural language supervision. In experiments, DietNeRF improves the perceptual quality of few-shot view synthesis when learned from scratch, can render novel views with as few as one observed image when pre-trained on a multi-view dataset, and produces plausible completions of completely unobserved regions.
References (20)
Learning Transferable Visual Models From Natural Language Supervision
Alec Radford, Jong Wook Kim, Chris Hallacy et al.
Scene Representation Networks: Continuous 3D-Structure-Aware Neural Scene Representations
V. Sitzmann, Michael Zollhoefer, Gordon Wetzstein
ImageNet: A large-scale hierarchical image database
Jia Deng, Wei Dong, R. Socher et al.
pixelNeRF: Neural Radiance Fields from One or Few Images
Alex Yu, Vickie Ye, Matthew Tancik et al.
Learning Transferable Visual Models From Natural Language Supervision
A. Linear-probe
Local light field fusion
B. Mildenhall, P. Srinivasan, Rodrigo Ortiz Cayon et al.
Representing Scenes as Neural Radiance Fields for View Synthesis
The Unreasonable Effectiveness of Deep Features as a Perceptual Metric
Richard Zhang, Phillip Isola, Alexei A. Efros et al.
Demystifying MMD GANs
Mikolaj Binkowski, Danica J. Sutherland, M. Arbel et al.
Ieee Transactions on Pattern Analysis and Machine Intelligence 1 What Shape Are Dolphins? Building 3d Morphable Models from 2d Images
T. Cashman, A. Fitzgibbon
Learning Category-Specific Mesh Reconstruction from Image Collections
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros et al.
GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
M. Heusel, Hubert Ramsauer, Thomas Unterthiner et al.
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency
Shubham Tulsiani, Tinghui Zhou, Alyosha A. Efros et al.
Image-to-Image Translation with Conditional Adversarial Networks
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou et al.
Structure-from-Motion Revisited
Johannes L. Schönberger, Jan-Michael Frahm
Training Deep Nets with Sublinear Memory Cost
Tianqi Chen, Bing Xu, Chiyuan Zhang et al.
GENERATIVE ADVERSARIAL NETS
Individualized Treat, Jinsung Yoon
Representation Learning: A Review and New Perspectives
Yoshua Bengio, Aaron C. Courville, P. Vincent
Ray tracing volume densities
J. Kajiya, B. V. Herzen
Cited By (20)
Node-RF: Learning Generalized Continuous Space-Time Scene Dynamics with Neural ODE-based NeRFs
\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathtt {M^3VIR}$$\end{document} for benchmarking sparse-view novel vi
LoD-Structured 3D Gaussian Splatting for Streaming Video Reconstruction
V3D: Enhancing text-to-3D synthesis through a view-consistent multi-view diffusion model
Novel view synthesis for underwater scenes with Gaussian splat fields and physically-based water modeling
PELR-GS: perception-enhanced large-scale 3D reconstruction for view-adaptive rendering
Illumination Refinement via Textual Cues: A Prompt-Driven Approach for Low-Light NeRF Enhancement
BP-NeRF: End-to-End Neural Radiance Fields for Sparse Images Without Camera Pose in Complex Scenes
Towards 3D Scene Understanding of Gas Plumes in LWIR Hyperspectral Images Using Neural Radiance Fields
GDoFS: Gaussian DoF Separation for Plausible 3D Geometry in Sparse-View 3DGS
Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting
Neural 4D Scene Reconstruction with Multiple One-Shot Scanning Systems
OrbitNVS: Harnessing Video Diffusion Priors for Novel View Synthesis
Energy-Efficient 3D Scene Representation: A Review of Sparse-View Gaussian Splatting for Sustainable AI
Consistency guided multiple plane image construction for novel view synthesis
DOC-GS: Dual-Domain Observation and Calibration for Reliable Sparse-View Gaussian Splatting
GeoRect4D: Geometry-Compatible Generative Rectification for Dynamic Sparse-View 3D Reconstruction
GuideGS: enhancing 3D Gaussian splatting with vision foundation models for sparse-view synthesis
Canvas3D: Translating 2D Edits into Implicit Neural Instance Field
First Shape, Then Meaning: Efficient Geometry and Semantics Learning for Indoor Reconstruction