PASR: Pose-Aware 3D Shape Retrieval from Occluded Single Views

As computer vision technology evolves, single-view 3D shape retrieval has become an important research direction. Traditional methods often rely on large-scale multimodal alignment, aligning 3D shape features with existing image-text embedding spaces. However, these methods have limited generalization capabilities when dealing with real-world images, as available 3D models are often not exact instance-level matches for 2D images. Additionally, existing methods typically learn holistic, global embeddings rather than explicit 3D geometry representations. This view-agnostic design is inherently vulnerable to partial occlusions and limits generalization to unseen 3D mesh models.

The core problem of single-view 3D shape retrieval is how to retrieve a corresponding 3D mesh given only a single RGB image. As large-scale 3D data becomes increasingly common, the need for effective retrieval methods becomes significant. Existing methods often perform poorly when handling partial occlusions and fine-grained geometric details, limiting their robustness and generalization to real-world applications.

PASR's core innovation lies in redefining 3D shape retrieval as an analysis-by-synthesis problem. First, the method distills fine-grained knowledge from a 2D foundation model, DINOv3, into a 3D encoder, achieving strong generalization capabilities to novel mesh shapes. Second, PASR aligns pose-conditioned 3D projections with 2D feature maps, bridging the gap between real-world images and synthetic meshes. Finally, during inference, PASR performs test-time optimization via analysis-by-synthesis, jointly searching for the shape and pose that best reconstruct the input image's feature map.

• �� During training, extract fine-grained knowledge from a 2D foundation model, DINOv3, and inject it into a 3D encoder.
• �� Align pose-conditioned 3D projections with 2D feature maps, bridging the gap between real-world images and synthetic meshes.
• �� During inference, perform test-time optimization via analysis-by-synthesis, jointly searching for the shape and pose that best reconstruct the input image's feature map.
• �� Use a differentiable renderer to project 3D features into the 2D feature space, aligning according to a given camera pose.

Experiments were conducted on the Pix3D and Pascal3D datasets, using Top-1 retrieval accuracy as the primary evaluation metric. Baseline models included CMIC, SC-IBSR, OpenShape, and Uni3D. The experimental design considered different levels of occlusion (L0-L3) to evaluate the model's robustness in complex scenarios. Results showed that PASR outperformed existing methods under all occlusion conditions, particularly excelling under L3 occlusion.

On the Pix3D dataset, PASR achieved an 81.59% Top-1 retrieval accuracy, showing an average relative improvement of 11.09% over previous best baseline methods. On the Pascal3D dataset, PASR achieved a 76.43% Top-1 retrieval accuracy, with an average relative improvement of 7.15% over previous best baseline methods. PASR demonstrates strong performance under occlusion scenarios, maintaining an accuracy of 63.05% under L3 occlusion conditions.

PASR has broad application potential in fields such as autonomous driving, robotic navigation, augmented reality, and virtual reality. In these applications, accurate 3D shape retrieval and pose estimation are crucial for efficient environmental perception and interaction. PASR offers a novel approach to enhancing accuracy and robustness in 3D shape retrieval, particularly in handling partial occlusions and fine-grained geometric details.

PASR may experience performance degradation under extreme occlusion or complex backgrounds, as these situations can complicate feature alignment. The method requires pose optimization during inference, which may increase computational overhead, particularly when handling large-scale datasets. In some cases, higher-quality 3D model databases may be needed to achieve optimal performance.