PUF: Plug-and-Play Uncertainty-Aware Fusion for Online 3D Scene Graph Generation
PUF framework improves relationship recall by 18.1% on 3DSSG while maintaining real-time performance.
Key Findings
Methodology
The PUF framework incrementally fuses 2D observations into a 3D scene graph using an uncertainty-aware probabilistic association method. Core components include Dirichlet evidence accumulation and an optional class-conditional prior, supporting both 3D Gaussian and voxel backends.
Key Results
- On the 3DSSG benchmark, the PUF-Gaussian method achieves a relationship recall of 46.0%, improving over FROSS by 18.1 points, with a processing time of only 15 ms per frame.
- On ReplicaSSG, PUF-Gaussian outperforms FROSS even without a relationship prior, demonstrating the framework's inherent reliability.
- The PUF-Voxel variant shows consistent improvements across both benchmarks, confirming the framework's representation agnosticism.
Significance
The PUF framework significantly enhances the accuracy and real-time capability of online 3D scene understanding by introducing an uncertainty-aware fusion method. It addresses the overlooked uncertainties in observations, models, and representations, offering a new paradigm for 3D scene graph generation in academia and industry.
Technical Contribution
Technically, the PUF framework replaces hard decisions with probabilistic associations, preserving 2D model uncertainty and propagating it into 3D reconstruction. It introduces a Dirichlet evidence accumulation mechanism, supports multiple 3D representations, and opens new engineering possibilities.
Novelty
PUF is the first framework to introduce uncertainty-aware fusion in 3D scene graph generation, providing a more flexible and accurate fusion mechanism compared to existing methods.
Limitations
- In scenarios lacking training data, the effectiveness of the class-conditional prior is limited, potentially leading to inaccurate relationship predictions.
- Probabilistic association may be less robust for extremely sparse or noisy data.
Future Work
Future work could explore more complex 3D representations and more efficient uncertainty modeling methods to further enhance the framework's applicability and performance.
AI Executive Summary
Current online 3D scene graph generation methods often overlook uncertainties in observations, models, and representations, limiting their application in complex scenes. To address this, researchers have proposed the PUF framework, which significantly enhances the accuracy and real-time capability of online 3D scene understanding by introducing an uncertainty-aware fusion method.
The core of the PUF framework lies in incrementally fusing 2D observations into a 3D scene graph using an uncertainty-aware probabilistic association method. Key components include Dirichlet evidence accumulation and an optional class-conditional prior, supporting both 3D Gaussian and voxel backends. This approach not only preserves 2D model uncertainty but also replaces traditional hard decisions with probabilistic associations.
Experimental results show that the PUF framework significantly outperforms existing methods on the 3DSSG and ReplicaSSG benchmarks. Notably, on 3DSSG, the PUF-Gaussian method achieves a relationship recall of 46.0%, improving over FROSS by 18.1 points, with a processing time of only 15 ms per frame.
Deep Dive
Abstract
Online 3D scene graph generation builds a persistent, structured representation of a scene by incrementally fusing 2D observations into a global 3D graph. Existing online methods treat this fusion as a fully deterministic pipeline, where we identify three sources of uncertainty that are overlooked: observation, 2D model, and 3D representation. We propose PUF: a Plug-and-play, Uncertainty-aware, and training-free Fusion framework. Scene graph node association is reformulated as a probabilistic likelihood over semantic and spatial factors, replacing binary accept/reject gates. Dirichlet evidence accumulation distributes class and relationship evidence across plausible candidates proportional to association likelihood. An optional class-conditional prior completes edges for sparsely or never co-observed object pairs. We instantiate PUF with both a 3D Gaussian and a 3D voxel backend and observe consistent improvements, demonstrating its ability to generalize across different representations. Experiments on the 3DSSG and ReplicaSSG benchmarks show that our method substantially outperforms existing approaches while maintaining real-time latency. These results establish uncertainty-aware fusion as a principled and effective paradigm for online 3D scene understanding. The source code is publicly available at https://github.com/yyyyangyi/PUF.