OCH3R: Object-Centric Holistic 3D Reconstruction

Recent years have seen rapid progress in single-image 3D scene understanding, with methods ranging from voxel-based models to neural radiance fields. High-capacity models like MeshCNN and NeRF variants have achieved impressive results but often require multiple views or extensive optimization, limiting real-time deployment. Gaussian-based methods, such as Gaussian Splatting, have enabled fast rendering and inference, yet they typically depend on external segmentation or per-object supervision, restricting scalability. Multi-stage pipelines combining detection, segmentation, and optimization are accurate but computationally expensive and fragile to errors, especially in cluttered scenes. Recent efforts aim to unify these steps into end-to-end frameworks, but achieving both speed and high fidelity remains challenging. This paper addresses this gap by proposing a Transformer-based, unified approach that predicts dense pixel attributes for all objects simultaneously, leveraging large-scale datasets and novel supervision strategies to improve robustness and efficiency.

The core challenge lies in reconstructing multiple objects and their poses from a single RGB image efficiently and accurately. Existing methods either rely on multi-stage pipelines, which are slow and error-prone, or scene-level models that lack instance-level detail. Achieving real-time, high-fidelity, object-centric 3D reconstruction in cluttered environments with occlusion and partial visibility remains unsolved. The difficulty is compounded by the need for precise pose estimation, shape completion, and semantic understanding, all from limited input data. Overcoming these bottlenecks is crucial for advancing applications in robotics, AR, and scene editing, where fast and reliable scene comprehension is essential.

The key innovations include: 1) a Transformer architecture that predicts dense, pixel-aligned attributes—category embeddings, depth, NOCS, and Gaussian primitives—in a single pass; 2) a Canonical Space Supervision framework that transforms predicted Gaussians into a standard reference frame, enabling supervision via pre-rendered multi-view images without explicit per-image labels; 3) an instance clustering and pose estimation pipeline that leverages predicted attributes for accurate object detection and alignment; 4) a unified training dataset combining multiple large-scale sources to enhance generalization. These innovations collectively enable fast, robust, and detailed multi-object 3D scene reconstruction from a single image, surpassing prior multi-stage and scene-level approaches.

• �� Input a single RGB image, extract dense features via DINOv2 backbone; • Use a 48-layer Transformer encoder to process features with global self-attention; • Predict per-pixel semantic embeddings (CLIP), depth, NOCS coordinates, and Gaussian parameters; • Apply a CRF to refine semantic affinities, producing instance masks; • Cluster pixels based on embeddings, estimate each object’s 6D pose using RANSAC-Umeyama with NOCS correspondences; • Transform predicted Gaussians into canonical space using estimated poses; • Render objects in canonical space with differentiable rasterizer, supervised against pre-rendered multi-view images; • Train with joint loss functions, including depth, segmentation, pose, and CSS loss, to enforce shape and pose consistency.

The model was evaluated on datasets including YCB-Video, HOPE, NOCS-Real, and PACE, using metrics like Chamfer Distance, F-1 score, and CLIP similarity. Baselines included Gen3DSR, AoE, and multi-stage pipelines. Results showed OCH3R achieved CD of 0.07, F-1 of 76.77, and inference time of 0.7s, outperforming baselines by large margins. Ablation studies confirmed the importance of Canonical Space Supervision and dense pixel prediction. The model maintained high performance in cluttered, occluded scenes, demonstrating robustness and scalability.

Quantitative comparisons revealed that OCH3R consistently outperformed existing methods, reducing Chamfer Distance by over 50% and increasing F-1 scores by more than 30 points across datasets. The inference speed was over 2000x faster than prior methods like Gen3DSR, enabling real-time scene understanding. Shape reconstructions were more complete and accurate, with high semantic consistency validated by CLIP scores. Ablation experiments highlighted the critical role of CSS in achieving shape completeness and robustness, especially under occlusion. These results establish OCH3R as a new state-of-the-art for single-image object-centric 3D reconstruction.

This technology can be immediately applied in robotic manipulation, where fast and accurate object detection and pose estimation are critical. It also benefits AR/VR content creation, enabling real-time scene editing and interaction. In autonomous systems, it provides rapid scene understanding for navigation and object handling. Long-term, integrating multi-modal data and extending to dynamic scenes could revolutionize scene perception, making machines capable of understanding and interacting with complex environments in real time.

Despite its strengths, the model faces challenges with extreme occlusion and highly symmetric objects, where pixel cues are insufficient for precise pose and shape recovery. Large-scale training data may introduce biases, affecting generalization to unseen categories. Computational demands, though optimized, still pose constraints for deployment on resource-limited devices, especially in real-time scenarios. Future work should address these issues by incorporating additional modalities, improving robustness to occlusion, and optimizing model efficiency.