Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactions
Proposes Gaussian-HOI optimizer for 3D human-object interaction reconstruction; introduces Open3DHOI dataset with 133 object categories and 120 interactions.
Key Findings
Methodology
This work combines single-image 3D reconstruction with Gaussian point cloud rendering to develop the Gaussian-HOI optimizer, enabling efficient spatial relationship learning. Using InstantMesh and OSX for initial coarse models, manual refinement improves interaction accuracy. The open-vocabulary dataset Open3DHOI, with 2.5k+ images and detailed annotations, supports diverse real-world scenarios. The unsupervised algorithm leverages Gaussian rendering to learn contact regions, significantly improving 3D HOI reconstruction quality in unconstrained environments.
Key Results
- On the Open3DHOI test set, the proposed method outperforms baseline PHOSA in object pose accuracy, reducing rotation error to 0.41, and significantly improving collision and contact metrics, with Co2 scores increasing by over 20%.
- The Gaussian-HOI optimizer enhances robustness across multiple object categories and complex poses, demonstrating superior generalization in wild scenes.
- Experimental data confirms that the method achieves high-fidelity 3D reconstructions with accurate spatial relations and contact regions, surpassing existing indoor-limited datasets.
Significance
This research addresses the critical gap in 3D HOI datasets for wild environments, enabling more realistic scene understanding. By integrating Gaussian point cloud rendering, it enhances the robustness and accuracy of 3D human-object modeling in complex scenarios, facilitating applications in robotics, AR, and scene analysis. The open-source dataset and unsupervised approach democratize access to diverse, real-world data, accelerating progress in 3D scene understanding and interaction modeling.
Technical Contribution
The core innovation is the Gaussian-HOI optimizer, which utilizes Gaussian point cloud rendering for joint optimization of human and object poses without training. The construction of the large-scale Open3DHOI dataset, covering extensive object categories and interactions, provides a valuable resource. The method combines multi-view coarse reconstruction with manual refinement, and the optimization leverages contact-aware Gaussian rendering, enabling high-precision, open-vocabulary 3D HOI reconstruction in unconstrained scenes.
Novelty
This is the first work to perform large-scale, open-vocabulary 3D human-object interaction reconstruction in wild scenes, using Gaussian point cloud rendering for contact and spatial relation learning. Unlike prior indoor-focused datasets and methods, it handles diverse, real-world objects and poses without reliance on predefined templates or training, representing a significant step forward in scene understanding.
Limitations
- The approach struggles with extreme occlusions and highly cluttered scenes, where contact regions are ambiguous or occluded.
- Reconstruction speed remains slower than traditional silhouette-based methods, limiting real-time applications.
- Heavy dependence on manual annotation for fine details and contact regions reduces automation and scalability.
Future Work
Future efforts will incorporate self-supervised learning to improve robustness against occlusion and clutter, aiming for real-time performance. Expanding the dataset with dynamic scenes and multi-modal data will further enhance model generalization. Integrating temporal information and multi-view cues could enable full scene understanding and interaction prediction in complex environments.
AI Executive Summary
Understanding human-object interactions in the wild is a longstanding challenge in computer vision, crucial for applications ranging from robotics to augmented reality. Existing datasets and methods predominantly focus on indoor scenes with limited object categories, restricting their applicability to real-world, diverse environments. To overcome this, the present work introduces a novel framework that leverages single-image 3D reconstruction combined with Gaussian point cloud rendering to achieve high-fidelity modeling of human-object interactions in unconstrained scenes.
At the core of this approach is the Gaussian-HOI optimizer, which efficiently learns the spatial relationships and contact regions between humans and objects without requiring training data. This is achieved by representing humans and objects as Gaussian point clouds, enabling differentiable rendering and contact inference. The method integrates multi-stage coarse reconstruction with manual annotation refinement, ensuring detailed and accurate 3D models.
Complementing the technical innovation, the authors constructed Open3DHOI, the first large-scale, open-vocabulary 3D HOI dataset in wild scenes. It contains over 2,500 images with annotations spanning 133 object categories and 120 interaction types, vastly surpassing prior indoor datasets in diversity and realism. Extensive experiments demonstrate that the proposed method outperforms baseline approaches like PHOSA, especially in complex, real-world scenarios, with significant improvements in object pose accuracy, collision detection, and contact region fidelity.
This work marks a significant advance in 3D scene understanding, bridging the gap between controlled indoor datasets and the unpredictable complexity of real-world environments. It opens new avenues for research in autonomous agents, AR, and scene analysis, emphasizing the importance of open-vocabulary, large-scale, real-world data. Despite current limitations in occlusion handling and computational efficiency, future directions include integrating self-supervised learning and multi-view data to further enhance robustness and scalability, promising a transformative impact on 3D interaction modeling.
Deep Dive
Abstract
Reconstructing human-object interactions (HOI) from single images is fundamental in computer vision. Existing methods are primarily trained and tested on indoor scenes due to the lack of 3D data, particularly constrained by the object variety, making it challenging to generalize to real-world scenes with a wide range of objects. The limitations of previous 3D HOI datasets were primarily due to the difficulty in acquiring 3D object assets. However, with the development of 3D reconstruction from single images, recently it has become possible to reconstruct various objects from 2D HOI images. We therefore propose a pipeline for annotating fine-grained 3D humans, objects, and their interactions from single images. We annotated 2.5k+ 3D HOI assets from existing 2D HOI datasets and built the first open-vocabulary in-the-wild 3D HOI dataset Open3DHOI, to serve as a future test set. Moreover, we design a novel Gaussian-HOI optimizer, which efficiently reconstructs the spatial interactions between humans and objects while learning the contact regions. Besides the 3D HOI reconstruction, we also propose several new tasks for 3D HOI understanding to pave the way for future work. Data and code will be publicly available at https://wenboran2002.github.io/3dhoi.