Shape-of-You: Fused Gromov-Wasserstein Optimal Transport for Semantic Correspondence in-the-Wild

Semantic correspondence has evolved from early handcrafted feature-based methods to deep learning approaches utilizing dense features from foundation models like DINO and CLIP. These models significantly improved zero-shot matching but still faced challenges in complex, real-world scenes with symmetries, occlusions, and repetitive patterns. Incorporating geometric information, especially 3D priors, has been shown to improve robustness, but existing methods often rely on explicit annotations like camera poses or 3D models, which are costly. Gromov-Wasserstein (GW) optimal transport has been effective in capturing structural similarities in point clouds and graphs, but its computational complexity limits practical use in high-dimensional feature spaces. This work bridges these gaps by integrating 3D geometric priors with FGW optimization, enabling structure-aware, unsupervised semantic correspondence without explicit annotations.

The core challenge is to establish accurate pixel-level correspondences across diverse, in-the-wild images without relying on explicit geometric annotations. Existing methods struggle with ambiguities caused by symmetries and repetitive features, as local feature matching in 2D appearance space is insufficient. Incorporating 3D geometric priors can mitigate these issues but introduces computational complexity. The problem is to develop a scalable, robust framework that jointly optimizes feature similarity and structural consistency in a computationally feasible manner, addressing the limitations of local matching and 2D-only approaches.

The paper's key innovations include: 1) Formulating semantic correspondence as a fused Gromov-Wasserstein (FGW) optimal transport problem, combining feature similarity with structural consistency; 2) Leveraging pretrained 3D models to define intra-structure in geometric space, resolving ambiguities from symmetries; 3) Developing an anchor-based linearization algorithm to approximate GW efficiently, enabling end-to-end training; 4) Introducing a soft target loss that blends guidance from probabilistic transport plans with network predictions, improving robustness to noise. These innovations collectively enable structure-aware, unsupervised matching that surpasses prior methods relying solely on appearance features.

• �� Extract dense semantic features from pretrained models (e.g., DINO) for each image region.
• �� Lift 2D image patches into 3D space using a pretrained 3D foundation model (VGGT), obtaining geometric coordinates.
• �� Initialize pseudo-labels via unbalanced optimal transport based on feature similarity, accommodating partial overlaps.
• �� Select high-confidence anchor pairs from initial transport plan, compute intra-instance distance matrices in 3D.
• �� Approximate Gromov-Wasserstein distance by linearizing around anchors, reducing quadratic complexity.
• �� Fuse semantic feature cost with geometric structure cost into a combined cost matrix.
• �� Iteratively optimize the transport plan, updating anchors and costs, to enforce global structural consistency.
• �� Use the resulting probabilistic plan as a soft supervision signal, blending with network predictions via a dynamic soft target loss.
• �� Train a lightweight adapter network to predict correspondences efficiently at inference.

The framework is evaluated on SPair-71k and AP-10k datasets, measuring [email protected]. Baselines include ASIC, DINOv2+SD, and DistillDIFT. Hyperparameters such as the number of anchors and the fusion weight α are tuned via ablation studies. The experiments demonstrate that incorporating geometric priors improves accuracy across categories, especially in ambiguous cases. The robustness of the soft target loss is validated through stability tests, and the efficiency of the anchor-based linearization is confirmed by runtime analysis. Cross-scenario tests show strong generalization, confirming the framework's scalability and effectiveness.

SoY achieves [email protected] of 67.9% on SPair-71k, outperforming previous methods (ASIC 39.9%, DINOv2+SD 63.5%). Incorporating the GW term improves accuracy by 2.3% to 7.0%, especially in symmetric and ambiguous cases. On AP-10k, performance gains are consistent across intra- and cross-species settings. Ablation results highlight the importance of geometric structure integration and anchor selection. The soft target loss enhances training stability, leading to better generalization and robustness against pseudo-label noise.

This method can be applied in automatic image annotation, 3D reconstruction, robotic perception, and augmented reality, where accurate semantic correspondence is crucial. Its unsupervised nature reduces reliance on costly annotations, making it suitable for large-scale deployment. The structure-aware approach improves robustness in real-world scenarios with complex geometries, occlusions, and symmetries. Future extensions could include multi-view integration, dynamic scene understanding, and multi-modal data fusion, broadening its industrial impact.

Despite its strengths, the approach faces challenges in scenes with severe occlusion or missing geometry, where geometric cues are insufficient. The reliance on pretrained 3D models increases computational costs, limiting real-time applications. Anchor-based linearization may oversimplify highly non-rigid deformations, affecting structure preservation. Scalability to very large scenes or cluttered environments remains limited, requiring further algorithmic optimization. Future work should focus on reducing complexity and enhancing robustness in diverse conditions.