Do It Yourself: Learning Semantic Correspondence from Pseudo-Labels
Proposes 3D-aware pseudo-label learning for semantic correspondence, achieving over 4% improvement on SPair-71k.
Key Findings
Methodology
This paper introduces a self-supervised framework leveraging 3D-aware pseudo-labels to enhance foundation model features for semantic correspondence. It involves generating pseudo-labels via zero-shot matching, propagating labels through multi-view chaining, and filtering with relaxed cyclic consistency and spherical geometric priors. These pseudo-labels supervise training of a lightweight adapter, refining features without manual keypoint annotations. The approach scales to large datasets like ImageNet-3D, improving generalization and robustness across diverse object categories and viewpoints.
Key Results
- On SPair-71k, the method achieves 77.2% [email protected], surpassing previous SOTA (74.6%) by over 4%. Pretraining on ImageNet-3D boosts performance to 78.6%. The method maintains strong results across categories and large viewpoint variations, demonstrating robustness.
- Multi-view chaining and geometric filtering significantly reduce false matches, improving pseudo-label quality. The relaxed cyclic consistency effectively filters out incompatible pairs, especially for symmetric or repetitive structures.
- Scaling to larger datasets like ImageNet-3D shows consistent performance gains, validating the approach's scalability. The method reduces reliance on manual annotations, enabling broader application in real-world scenarios.
Significance
This work addresses the long-standing challenge of semantic correspondence without extensive manual labeling. By integrating 3D geometric cues into pseudo-label generation, it enhances the robustness and accuracy of feature matching across diverse instances. The approach bridges the gap between foundation models and practical large-scale applications, facilitating advances in areas like autonomous driving, robotics, and AR/VR. Its scalability and minimal annotation requirement make it highly impactful for both academia and industry, pushing the frontier of unsupervised semantic understanding.
Technical Contribution
Key innovations include: 1) zero-shot pseudo-label generation using foundation features; 2) multi-view chain propagation with relaxed cyclic consistency to improve label quality; 3) spherical geometric filtering to reject false positives; 4) supervised training of a lightweight adapter with dense and sparse losses. These components collectively improve feature discriminability and geometric consistency, surpassing prior weakly supervised methods, and enabling scalable, annotation-free training.
Novelty
This is the first comprehensive integration of 3D-aware pseudo-labeling with multi-view chaining, cyclic consistency, and geometric filtering for semantic correspondence. Unlike prior works relying on manual keypoints or category-specific priors, this approach leverages large-scale 3D pose data and foundation model features to generate high-quality pseudo labels automatically. It significantly advances the state-of-the-art in unsupervised semantic matching, especially for complex, symmetric, and articulated objects.
Limitations
- The method's performance declines with extreme viewpoint changes or heavy occlusion, where pseudo-labels become less reliable. It still depends on the quality of initial foundation features and 3D pose annotations.
- Computational costs are high due to multi-view chaining and filtering steps, limiting real-time deployment. Further optimization is needed for efficiency.
- Handling highly non-rigid or deformable objects remains challenging, as the geometric priors assume rigid structures. Extending to dynamic scenes is future work.
Future Work
Future directions include integrating multi-modal cues (e.g., depth, motion), improving pseudo-label quality in challenging scenarios, reducing computational overhead, and extending to non-rigid and dynamic objects. Exploring self-supervised learning with less reliance on 3D annotations and applying the framework to video sequences are promising avenues.
AI Executive Summary
Semantic correspondence—the task of matching semantically similar points across different images—has long been a core challenge in computer vision. Traditional methods relied heavily on manual annotations or handcrafted features, limiting scalability and robustness. Recent advances in foundation models like DINO and diffusion-based features have shown promise, but ambiguities remain, especially with symmetric objects and large viewpoint variations. To address these issues, this work introduces a novel self-supervised framework that leverages 3D-aware pseudo-labels, generated through zero-shot matching, multi-view chaining, and geometric filtering. This approach significantly reduces the need for manual keypoint annotations, enabling scalable training on large datasets such as ImageNet-3D. The core idea is to refine foundation features by supervising a lightweight adapter with high-quality pseudo-labels, which are filtered via relaxed cyclic consistency and spherical geometric priors. Extensive experiments on SPair-71k demonstrate that the proposed method achieves 77.2% [email protected], outperforming previous state-of-the-art methods by over 4%. The performance gains are consistent across categories and viewpoints, validating the robustness and generalization of the approach. Moreover, scaling to larger datasets further improves results, confirming its scalability. This work marks a significant step toward unsupervised, scalable semantic matching, with broad implications for applications in autonomous driving, robotics, and AR/VR. Despite its strengths, challenges remain in handling extreme occlusion and non-rigid objects, and future work will focus on multi-modal integration, efficiency improvements, and extension to dynamic scenes.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Finding correspondences between semantically similar points across images and object instances is one of the everlasting challenges in computer vision. While large pre-trained vision models have recently been demonstrated as effective priors for semantic matching, they still suffer from ambiguities for symmetric objects or repeated object parts. We propose improving semantic correspondence estimation through 3D-aware pseudo-labeling. Specifically, we train an adapter to refine off-the-shelf features using pseudo-labels obtained via 3D-aware chaining, filtering wrong labels through relaxed cyclic consistency, and 3D spherical prototype mapping constraints. While reducing the need for dataset-specific annotations compared to prior work, we establish a new state-of-the-art on SPair-71k, achieving an absolute gain of over 4% and of over 7% compared to methods with similar supervision requirements. The generality of our proposed approach simplifies the extension of training to other data sources, which we demonstrate in our experiments.