Real-time Simultaneous Multi-Object 3D Shape Reconstruction, 6DoF Pose Estimation and Dense Grasp Prediction
SceneGrasp employs deep learning for real-time multi-object 3D reconstruction, 6DoF pose estimation, and dense grasp prediction from a single RGBD image.
Key Findings
Methodology
SceneGrasp integrates ResNet-FPN backbone with heatmap-based object center detection, combined with ScaleShapeGrasp-AE for joint shape and grasp encoding. The scene understanding network (SceneGrasp-Net) predicts object categories, 6DoF poses, scales, and dense grasp parameters via learned embeddings. Training involves a two-stage process: first, optimizing the auto-encoder for shape and grasp in a canonical object frame; second, training the scene network for semantic and pose estimation. During inference, object centers are localized via heatmaps, and embeddings are decoded to produce full 3D shapes, poses, and grasp points, which are then transformed into the camera frame for manipulation tasks. The entire pipeline achieves 30FPS, enabling real-time multi-object perception.
Key Results
- On the NOCS dataset, our method surpasses NOCS in 3D detection and pose accuracy, with an average positional error of ~3cm and rotational error below 5°. Shape reconstruction via Chamfer distance reaches 1.8cm, comparable to CenterSnap. Grasp success rates exceed 78% across various coverage thresholds, with inference speed at 30FPS, outperforming multi-stage pipelines in both accuracy and efficiency.
- In cluttered scenes, the model accurately detects and reconstructs multiple objects, including occluded and unobserved regions, demonstrating strong generalization. Ablation studies confirm that joint training of shape and grasp encodings enhances overall performance.
- The approach maintains high robustness under occlusion and partial observations, with ablation showing the importance of the auto-encoder and dense grasp prediction modules for accuracy.
Significance
This work advances robotic scene understanding by enabling simultaneous, real-time perception of multiple objects' geometry, pose, and graspability from a single RGBD view. It addresses the bottleneck of multi-stage processing, providing a unified framework suitable for reactive manipulation in cluttered environments. The high inference speed and accuracy facilitate deployment in industrial automation, logistics, and service robots, marking a significant step toward autonomous, perceptive robotic systems.
Technical Contribution
Key innovations include: 1) a ResNet-FPN based detection and embedding extraction for multi-object localization; 2) a scale-aware auto-encoder that learns joint shape and grasp representations in a canonical frame; 3) an end-to-end training strategy that jointly optimizes geometric, semantic, and grasp predictions, enabling fast inference. These contributions collectively improve accuracy, robustness, and computational efficiency over prior multi-stage or view-dependent methods.
Novelty
This is the first framework to perform simultaneous multi-object 3D reconstruction, 6DoF pose estimation, and dense grasp prediction from a single RGBD image in real-time. Unlike previous works that handle these tasks separately or rely on multi-view data, our method unifies them in a single feed-forward pipeline with explicit scale and pose modeling, pushing the boundary of scene understanding and manipulation.
Limitations
- The model's accuracy diminishes under extreme occlusion or when objects are very small, due to limited training data for such scenarios.
- Handling highly cluttered or dynamic scenes remains challenging, especially when multiple objects occlude each other.
- High-precision geometric reconstruction and dense grasp prediction demand significant computational resources, limiting deployment on low-end hardware.
Future Work
Future research will explore multi-view fusion, active perception, and reinforcement learning to improve robustness under occlusion and motion. Additionally, efforts will focus on model compression and hardware optimization to enable deployment on embedded systems, expanding applications in mobile robots and industrial automation.
AI Executive Summary
Robotic manipulation in complex environments demands rapid, accurate scene understanding to enable autonomous decision-making. Traditional pipelines often process perception and grasp planning sequentially, resulting in latency and compounded errors. To address this, we introduce SceneGrasp, a unified deep learning framework capable of real-time multi-object 3D reconstruction, 6DoF pose estimation, and dense grasp prediction from a single RGBD image.
SceneGrasp leverages a ResNet-FPN backbone to localize object centers via heatmaps, combined with a novel ScaleShapeGrasp auto-encoder that encodes object geometry and grasping parameters in a canonical space. The scene understanding network (SceneGrasp-Net) predicts object categories, poses, scales, and embeddings, which are decoded into full 3D shapes and dense grasp points. These are then transformed into the camera frame, facilitating immediate manipulation.
Experimental results on the NOCS dataset demonstrate that our approach outperforms existing methods like NOCS and CenterSnap in detection accuracy, shape reconstruction, and grasp success rate, while maintaining a high inference speed of 30FPS. The model achieves an average positional error of approximately 3cm and a rotational error below 5°, with shape Chamfer distances around 1.8cm, validating its precision.
This integrated approach significantly reduces perception-to-action latency, enabling robots to operate efficiently in cluttered, dynamic scenes. Its robustness under occlusion and partial observations marks a substantial step forward in autonomous scene understanding. Future work aims to incorporate multi-view data and reinforcement learning to further enhance performance, broadening the scope of robotic applications in industry and service sectors.
Deep Dive
Abstract
Robotic manipulation systems operating in complex environments rely on perception systems that provide information about the geometry (pose and 3D shape) of the objects in the scene along with other semantic information such as object labels. This information is then used for choosing the feasible grasps on relevant objects. In this paper, we present a novel method to provide this geometric and semantic information of all objects in the scene as well as feasible grasps on those objects simultaneously. The main advantage of our method is its speed as it avoids sequential perception and grasp planning steps. With detailed quantitative analysis, we show that our method delivers competitive performance compared to the state-of-the-art dedicated methods for object shape, pose, and grasp predictions while providing fast inference at 30 frames per second speed.