TensoRF: Tensorial Radiance Fields
TensoRF employs tensor decomposition of 4D feature tensors for fast, compact scene radiance field reconstruction, outperforming NeRF in speed and size.
Key Findings
Methodology
This work models the scene radiance field as a 4D tensor representing a feature grid, decomposed via CP and a novel vector-matrix (VM) approach. CP decomposes the tensor into a sum of rank-one outer products, suitable for simple scenes, enabling rapid training (<30 min) with minimal memory (<4MB). VM decomposition relaxes low-rank constraints by factorizing tensor modes into vectors and matrices, capturing complex scene details with fewer components, reducing training time (<10 min) and memory footprint (<75MB). The approach integrates explicit feature grids, supports view-dependent color via MLP or spherical harmonics, and employs differentiable volume rendering for high-quality synthesis.
Key Results
- CP-based TensoRF achieves scene reconstruction in under 30 minutes with PSNR of 31.95, model size below 4MB, outperforming NeRF (PSNR 31.01).
- VM-based TensoRF further reduces training time to under 10 minutes, with PSNR reaching 33.14, model size under 75MB, and superior rendering quality.
- Across synthetic and real datasets, the method demonstrates significant memory savings (O(n) to O(n^2)), faster convergence, and comparable or better visual fidelity than state-of-the-art methods.
Significance
This innovation addresses the core bottleneck in neural radiance field modeling—training speed and model size—by leveraging tensor low-rank structures. It enables large-scale scene reconstruction with minimal storage, facilitating real-time applications in AR/VR, virtual production, and robotics. The approach bridges the gap between high-fidelity rendering and computational efficiency, opening avenues for scalable 3D scene understanding.
Technical Contribution
The paper introduces a tensorial scene representation combining CP and VM decompositions, enabling explicit feature grid modeling with low-rank regularization. It supports continuous volume rendering through efficient trilinear interpolation of tensor factors. The method significantly reduces memory complexity from O(n^3) to O(n^2) or lower, while maintaining or surpassing the rendering quality of existing neural radiance fields. The framework is compatible with various appearance decoding functions, including neural networks and spherical harmonics.
Novelty
This is the first work to formulate scene radiance fields explicitly as 4D tensors decomposed via CP and VM methods, integrating tensor low-rank regularization into neural scene reconstruction. Unlike prior voxel-based or MLP-based models, TensoRF exploits tensor structures for both efficiency and expressiveness, representing a fundamental shift in scene modeling paradigms.
Limitations
- While efficient, the model may require many components for extremely complex scenes, which can marginally increase training time and complexity.
- Currently optimized for static scenes; dynamic scene extension remains future work.
- Performance may degrade with very sparse or noisy input views, affecting reconstruction fidelity.
Future Work
Future directions include extending the tensor decomposition framework to dynamic scenes, developing multi-scale and multi-resolution models, and incorporating cross-scene generalization techniques. Further research may explore adaptive tensor ranks and hybrid models combining neural and explicit representations for even better efficiency and quality.
AI Executive Summary
The advent of neural radiance fields (NeRF) revolutionized 3D scene reconstruction, enabling photorealistic view synthesis from multi-view images. However, traditional NeRF models rely on multilayer perceptrons (MLPs), which, despite their expressive power, suffer from slow training and large memory footprints, limiting their scalability. Addressing these challenges, this paper introduces TensoRF, a novel scene representation leveraging tensor decomposition techniques to model the scene as an explicit feature grid expressed as a 4D tensor.
TensoRF employs two core tensor factorization methods: the classical CANDECOMP/PARAFAC (CP) decomposition and a new vector-matrix (VM) decomposition. CP decomposes the scene tensor into a sum of rank-one outer products, providing a highly compact representation suitable for simple scenes. The VM approach relaxes the low-rank constraints by factorizing two modes into matrices, significantly enhancing the model's capacity to encode complex details with fewer components.
The key innovation lies in representing the scene as a feature grid, where each voxel's features are obtained via efficient trilinear interpolation of the tensor factors. This explicit representation supports continuous volume rendering, enabling high-fidelity view synthesis. The entire framework is optimized via gradient descent, with regularization to prevent overfitting, and supports diverse appearance decoding functions, including small MLPs and spherical harmonics.
Experimental results demonstrate that CP-based TensoRF can reconstruct scenes in under 30 minutes with a model size below 4MB, outperforming NeRF in PSNR and rendering quality. VM-based models further reduce training time to under 10 minutes, with PSNR exceeding 33 and model size under 75MB, while delivering superior visual fidelity. These results are consistent across synthetic and real datasets, showing remarkable memory efficiency and scalability.
Overall, TensoRF offers a transformative approach to scene modeling, combining the strengths of explicit feature grids and tensor low-rank structures. It achieves a compelling balance of speed, compactness, and quality, paving the way for real-time, large-scale 3D scene reconstruction in practical applications such as AR/VR, virtual production, and robotics. Future work will explore dynamic scene extensions, multi-resolution strategies, and cross-scene generalization, further broadening its impact.
Deep Dive
Abstract
We present TensoRF, a novel approach to model and reconstruct radiance fields. Unlike NeRF that purely uses MLPs, we model the radiance field of a scene as a 4D tensor, which represents a 3D voxel grid with per-voxel multi-channel features. Our central idea is to factorize the 4D scene tensor into multiple compact low-rank tensor components. We demonstrate that applying traditional CP decomposition -- that factorizes tensors into rank-one components with compact vectors -- in our framework leads to improvements over vanilla NeRF. To further boost performance, we introduce a novel vector-matrix (VM) decomposition that relaxes the low-rank constraints for two modes of a tensor and factorizes tensors into compact vector and matrix factors. Beyond superior rendering quality, our models with CP and VM decompositions lead to a significantly lower memory footprint in comparison to previous and concurrent works that directly optimize per-voxel features. Experimentally, we demonstrate that TensoRF with CP decomposition achieves fast reconstruction (<30 min) with better rendering quality and even a smaller model size (<4 MB) compared to NeRF. Moreover, TensoRF with VM decomposition further boosts rendering quality and outperforms previous state-of-the-art methods, while reducing the reconstruction time (<10 min) and retaining a compact model size (<75 MB).