SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

The evolution of multimodal learning has seen models like CLIP and ALIGN achieve impressive zero-shot performance by training on hundreds of millions of image-text pairs. These models rely heavily on contrastive learning objectives, which require extensive paired datasets, making training costly and limiting adaptation in data-scarce domains. Recent efforts have explored weakly supervised and semi-supervised methods, such as UOT-RCL, to reduce data dependency. However, these approaches still face scalability issues, especially in large-scale training involving optimal transport. The 'Platonic Representation Hypothesis' suggests that pretrained unimodal encoders inherently encode compatible semantic structures, motivating methods that keep encoders frozen and learn lightweight alignment layers. This paper builds on this idea, proposing a scalable semi-supervised framework that leverages minimal paired data and abundant unpaired data, addressing the bottlenecks of existing OT-based alignment techniques.

The core challenge is achieving accurate cross-modal alignment with very limited paired data, which is essential for applications where data annotation is expensive or infeasible. Traditional contrastive methods demand millions of pairs for optimal performance, while OT-based approaches struggle with computational scalability due to the high memory cost of Sinkhorn iterations. The problem becomes more complex when unpaired data from different distributions are introduced, risking misalignment and reduced robustness. Developing a method that efficiently propagates geometric and relational information from few pairs to large unpaired datasets remains an open challenge, crucial for democratizing multimodal AI in real-world scenarios.

The paper introduces several key innovations: 1) KLOT divergence, which leverages optimal transport to transfer relational structure efficiently, with explicit gradient derivation to enable large-batch training; 2) a two-stage approach where a simple linear teacher model is trained on limited pairs to recover coarse geometry, then used to regularize the training of deep alignment layers; 3) the integration of relation transfer mechanisms that preserve geometric relations across datasets; 4) a fully differentiable, memory-efficient gradient formula that overcomes the bottlenecks of traditional OT algorithms. These innovations collectively enable scalable, robust semi-supervised alignment, setting new standards in the field.

• �� Stage 1: Fit a linear alignment model (Procrustes, CCA, or contrastive) using limited paired samples to estimate projection matrices (Wx, Wy). Input: small paired dataset; Output: initial geometric structure.
• �� Stage 2: Use the linear model as a teacher to regularize the training of deep alignment layers (fθ1, gθ2). Input: large unpaired datasets; Process: minimize divergence between the model affinity matrix and the target geometry derived from the teacher.
• �� Core algorithms include cosine similarity for affinity computation, the KLOT divergence based on optimal transport with explicit gradient derivation, and contrastive loss functions.
• �� The training objective combines supervised loss on few pairs with a regularization term enforcing geometric consistency on unpaired data.
• �� The explicit gradient formula (Theorem 5.1) enables efficient backpropagation, avoiding memory bottlenecks.
• �� Hyperparameters such as regularization coefficient α and entropy parameters are tuned via validation on retrieval tasks.

• �� Datasets: COCO, CC3M, CC12M, with 10k paired samples and up to 1 million unpaired images/texts.
• �� Baselines: Procrustes, CCA, contrastive learning, and existing semi-supervised methods.
• �� Metrics: Zero-shot retrieval Mean R@1, cross-dataset generalization.
• �� Ablations: Effect of number of pairs, unpaired data, divergence choices, and gradient derivation.
• �� Hyperparameter tuning: Regularization weight α, entropy parameters ϵ, ϵ∗.
• �� Results show that with minimal supervision, the model surpasses traditional methods, especially when leveraging large unpaired datasets, demonstrating robustness across data sources and distribution shifts.

• �� SOTAlign achieves 30.0% Mean R@1 on COCO with only 10k pairs, outperforming SigLIP (24.2%) and other baselines.
• �� Incorporating over 1 million unpaired samples yields further performance gains, confirming the effectiveness of relation transfer.
• �� The explicit gradient approach accelerates training by 50×, enabling large-batch scalability.
• �� The method maintains robustness under distribution shifts, such as using data from ImageNet, CC12M, and synthetic captions, with performance improvements of up to 10% over supervised baselines.

• �� Enables low-cost, scalable vision-language alignment in medical imaging, scientific data analysis, and industrial automation.
• �� Facilitates cross-domain adaptation where annotated data is scarce, improving model robustness and generalization.
• �� Supports building multimodal systems with minimal labeled data, reducing reliance on costly annotation processes.

• �� Performance drops when unpaired data are from highly mismatched distributions, indicating sensitivity to data heterogeneity.
• �� Hyperparameter tuning remains complex, requiring extensive validation.
• �� High computational demands for large batch training, despite gradient improvements, still pose practical challenges.