DINeMo: Learning Neural Mesh Models with no 3D Annotations
DINeMo leverages large pretrained models for pseudo-correspondence, enabling zero-shot 3D pose estimation with 67.3% gap reduction to supervised methods.
Key Findings
Methodology
DINeMo adopts an analysis-by-synthesis framework, utilizing features from large-scale pretrained models like DINOv2 and Stable Diffusion to generate pseudo-correspondences across multiple views. The process involves extracting features, computing cosine similarities for initial matches, then applying a bidirectional refinement that incorporates global pose voting and local feature consistency. The training optimizes a part-contrastive loss on vertex features and pose parameters without requiring 3D annotations. During inference, the approach integrates Grounded-SAM masks to improve occlusion robustness. This methodology enables scalable, annotation-free 3D pose learning, leveraging the rich representations of foundation models for category generalization.
Key Results
- On Pascal3D+ car dataset, DINeMo achieves 92.8% accuracy at π/6 in zero-shot mode, outperforming 3D-DST (82.3%) and NVS (65.5%) with 50 shots, reducing the performance gap with fully supervised methods by 67.3%.
- On SPair71k semantic correspondence, DINeMo attains [email protected] of 59.1%, surpassing SD-DINO and Telling Left from Right, demonstrating strong cross-category matching capabilities.
- Scaling experiments show performance steadily improves as unlabeled data increases from 2,048 to 15,000 images, confirming the method’s efficiency and scalability with abundant unannotated data.
- In occlusion scenarios, DINeMo exhibits superior robustness, closely approaching fully supervised models, indicating practical viability in real-world applications.
Significance
This work addresses the critical bottleneck of reliance on costly 3D annotations for pose estimation. By exploiting large pretrained models for feature-based pseudo-labeling, it enables scalable, category-agnostic, and robust 3D understanding. The approach significantly broadens the applicability of neural mesh models, facilitating deployment in robotics, autonomous driving, and AR/VR, where annotated data is scarce. It also advances the paradigm of unsupervised 3D learning, opening avenues for future research in self-supervised visual understanding and large-scale 3D scene comprehension.
Technical Contribution
The key technical innovation lies in the bidirectional pseudo-correspondence generation, which combines local feature similarity with global pose consistency, derived from foundation model features. This mechanism enhances pseudo-label quality, enabling effective training without 3D supervision. The integration of analysis-by-synthesis with foundation model features and mask-based occlusion handling constitutes a novel framework, pushing the boundaries of unsupervised 3D pose estimation. The method’s scalability and robustness stem from leveraging rich, generalizable representations from large-scale pretrained models, offering a new pathway for annotation-free 3D understanding.
Novelty
This research is the first to utilize large foundation models for generating pseudo-correspondences in neural mesh training, bypassing the need for explicit 3D annotations. The bidirectional refinement process, combining local and global cues, significantly improves correspondence accuracy. Unlike prior methods relying on synthetic data or manual annotations, this approach harnesses pretrained feature spaces, enabling broad category generalization and robust performance in occlusion and domain shift scenarios. It represents a paradigm shift from supervised to self-supervised 3D learning frameworks.
Limitations
- Dependence on the quality and biases of pretrained foundation models may limit performance in certain categories or under specific conditions. The pseudo-correspondence noise can still cause errors, especially in extreme occlusion or highly similar objects.
- Computational cost remains high due to multi-view feature extraction and iterative optimization, posing challenges for real-time deployment.
- Current framework primarily targets rigid objects, and extending to deformable or articulated objects requires further adaptation.
Future Work
Future directions include integrating multi-modal data (e.g., depth, LiDAR) to enhance robustness, developing more efficient feature extraction and matching algorithms, and exploring self-supervised fine-tuning to reduce reliance on large foundation models. Extending the approach to articulated objects and dynamic scenes, as well as optimizing for real-time inference, are promising avenues to broaden practical deployment.
AI Executive Summary
Estimating the 3D pose of objects from 2D images is fundamental for scene understanding but traditionally relies heavily on expensive annotated datasets. This dependency limits scalability and generalization, especially across categories and under occlusion. Recent advances in neural mesh models and analysis-by-synthesis approaches have improved robustness but still depend on 3D supervision, constraining their broader application.
In this context, DINeMo introduces a groundbreaking approach that leverages large pretrained visual foundation models—such as DINOv2—to generate pseudo-correspondences across views. By employing a bidirectional matching process that combines local appearance features with global pose cues, the method refines initial noisy matches into high-quality pseudo-labels. These labels enable training neural mesh models without any 3D annotations, dramatically reducing data collection costs.
Experimental results on Pascal3D+ and SPair71k datasets demonstrate that DINeMo achieves state-of-the-art zero-shot performance, narrowing the gap with fully supervised models by over 67%. Its robustness to partial occlusion and domain shifts surpasses previous methods, making it highly suitable for real-world applications like autonomous driving and robotics.
Moreover, the approach scales effectively with increasing unlabeled data, showing continuous performance gains as more images are incorporated. This scalability, combined with the elimination of annotation bottlenecks, marks a significant step forward in unsupervised 3D understanding.
Despite these advances, challenges remain, including computational costs and handling deformable objects. Future work aims to incorporate multi-modal data, optimize efficiency, and extend to more complex scene types, promising a new era of scalable, annotation-free 3D scene understanding.
Deep Dive
Abstract
Category-level 3D/6D pose estimation is a crucial step towards comprehensive 3D scene understanding, which would enable a broad range of applications in robotics and embodied AI. Recent works explored neural mesh models that approach a range of 2D and 3D tasks from an analysis-by-synthesis perspective. Despite the largely enhanced robustness to partial occlusion and domain shifts, these methods depended heavily on 3D annotations for part-contrastive learning, which confines them to a narrow set of categories and hinders efficient scaling. In this work, we present DINeMo, a novel neural mesh model that is trained with no 3D annotations by leveraging pseudo-correspondence obtained from large visual foundation models. We adopt a bidirectional pseudo-correspondence generation method, which produce pseudo correspondence utilize both local appearance features and global context information. Experimental results on car datasets demonstrate that our DINeMo outperforms previous zero- and few-shot 3D pose estimation by a wide margin, narrowing the gap with fully-supervised methods by 67.3%. Our DINeMo also scales effectively and efficiently when incorporating more unlabeled images during training, which demonstrate the advantages over supervised learning methods that rely on 3D annotations. Our project page is available at https://analysis-by-synthesis.github.io/DINeMo/.