Same Concept, Different Directions: Cross-Modal Feature Heterogeneity in Sparse Autoencoders
Proposes modality-specific sparse autoencoders and post-hoc alignment to address cross-modal feature heterogeneity, boosting retrieval accuracy to 85.3% on MS-COCO.
Key Findings
Methodology
This study investigates the phenomenon of cross-modal feature heterogeneity in joint embedding spaces of vision-language models (VLMs). It introduces modality-specific sparse autoencoders (SAEs) trained independently on image and text embeddings to preserve their unique feature geometries. The core innovation involves computing correlation matrices between latent codes of paired image-text samples, then applying the Hungarian algorithm for optimal permutation to align features post hoc. This approach avoids the assumption that shared concepts have identical feature directions across modalities, instead allowing for intrinsic heterogeneity. Experiments on datasets like MS-COCO and Flickr30k validate the method, evaluating reconstruction fidelity, cross-modal retrieval, and concept steering metrics.
Key Results
- The proposed approach achieves a Recall@1 of 85.3% in cross-modal retrieval, outperforming previous alignment methods that averaged 78.6%, demonstrating improved semantic matching accuracy.
- In concept steering tasks, the model's ability to manipulate semantic concepts improved by 15%, indicating better fine-grained control over representations.
- Reconstruction error decreased by 12%, confirming that modality-specific encoding effectively preserves feature geometry, leading to more robust representations.
Significance
This work fundamentally challenges the prevalent assumption that shared semantic concepts must align along the same feature directions across modalities. By revealing the existence of feature heterogeneity, it offers a new perspective on the intrinsic differences in visual and textual representations. The methodology enhances the interpretability and controllability of joint embeddings, which are critical for applications like cross-modal retrieval, semantic editing, and multimodal generation. It bridges a crucial gap between theoretical understanding and practical deployment, paving the way for more flexible and accurate multimodal systems.
Technical Contribution
The paper's key technical contribution is the development of modality-specific SAEs combined with a post-hoc alignment strategy based on correlation matrices and the Hungarian algorithm. This approach preserves the geometric structure of each modality's features, avoiding the pitfalls of forced feature direction sharing. Theoretical analysis demonstrates how limited latent capacity leads to feature collapse, and the proposed method effectively mitigates this by maintaining modality-specific directions while aligning concepts. These innovations extend the linear representation hypothesis to a more realistic, heterogeneity-aware framework, providing formal guarantees on reconstruction and alignment performance.
Novelty
This research is the first to explicitly define and analyze cross-modal feature heterogeneity, moving beyond the traditional assumption of shared feature directions. It introduces a novel two-stage framework—training modality-specific SAEs followed by post-hoc alignment—that effectively handles the intrinsic differences in visual and textual feature spaces. Unlike prior methods that enforce strict feature sharing, this approach respects modality-specific geometries, leading to superior retrieval and interpretability results. Its combination of theoretical insights and practical algorithms marks a significant advancement in multimodal representation learning.
Limitations
- The method relies on pre-trained VLM features, which may limit generalization across different models or domains. Its effectiveness depends on the quality of initial embeddings.
- Correlation-based matching can be sensitive to noise and may struggle with ambiguous or low-quality data, affecting alignment accuracy.
- In scenarios with extreme modality divergence or very limited data, the heterogeneity may be more complex, requiring further robustness enhancements.
Future Work
Future directions include integrating dynamic feature modeling to capture temporal variations, exploring adversarial training to enhance feature geometry preservation, and extending the framework to multi-turn dialogue or multimodal generation tasks. Additionally, investigating unsupervised or weakly supervised alignment strategies could broaden applicability in real-world settings.
AI Executive Summary
In recent years, vision-language models (VLMs) such as CLIP and ALIGN have revolutionized multimodal understanding by embedding images and text into a shared space. Despite their success, these models face a fundamental challenge: the semantic features of concepts often do not align along the same directions across modalities. This phenomenon, termed cross-modal feature heterogeneity, manifests as the same concept activating different latent codes depending on whether it is represented visually or textually. Such heterogeneity leads to the well-known modality split, where the shared concept is fragmented across different regions of the embedding space, hampering tasks like cross-modal retrieval and concept manipulation.
Traditional approaches attempted to align these features by forcing shared latent representations, assuming that the semantic features share the same directions. However, this assumption overlooks the intrinsic differences in visual and textual representations. To address this, the authors propose a novel framework that trains separate sparse autoencoders (SAEs) for image and text embeddings, preserving their unique feature geometries. The core idea is to compute correlation matrices between the latent codes of paired samples and then apply the Hungarian algorithm for optimal permutation, aligning features post hoc without forcing directional similarity.
This method effectively maintains the geometric integrity of each modality's features while achieving meaningful alignment at the concept level. Extensive experiments on datasets like MS-COCO and Flickr30k demonstrate that this approach significantly improves cross-modal retrieval accuracy, achieving a Recall@1 of 85.3%, surpassing previous methods. Additionally, the model shows enhanced capability in concept steering, allowing finer semantic control. Theoretical analysis confirms that limited latent capacity naturally causes feature collapse, and the proposed post-hoc alignment mitigates this without sacrificing reconstruction quality.
Overall, this work shifts the paradigm from enforcing feature direction sharing to respecting modality-specific geometries, unlocking more accurate and interpretable multimodal representations. It opens new avenues for research in multimodal understanding, generation, and controllable AI, with promising implications for future intelligent systems. Limitations include reliance on pre-trained features and sensitivity to correlation estimation, suggesting directions for further robustness and generalization improvements.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房里准备一顿饭，不同厨师用不同的工具和方法做菜。有的用刀切菜，有的用剪刀，但他们都在做同一道菜。以前的想法是让所有厨师用一样的工具，结果反而限制了他们的发挥。现在，聪明的厨师会根据自己的习惯用不同的工具，然后用一种特别的方法找到他们的共同点，让每个人都能发挥出最好的效果。这样，厨房里的每个人都能做出美味的菜肴，而不需要强求他们用一样的工具。这就像在多模态学习中，保持每个模态的特色，同时找到它们的联系，才能做出更聪明、更灵活的系统。
ELI14 Explained like you're 14
你知道在学校里，有时候你和朋友都在讲同一个故事，但每个人讲的角度不一样。有的用手势，有的用表情，但其实都在讲同一件事。以前的老师会说，要让你们讲得一样，用相同的手势和表情，但这样其实限制了每个人的表达。现在，老师建议你们各自用自己的方式讲，然后用一种特别的办法，把你们的表达联系起来。这样，不仅每个人都能自由表达，还能找到共同点，让大家都明白对方在讲什么。这个故事告诉我们，在多模态学习中，保持每种表达方式的特色，同时找到它们的联系，才是最聪明的办法。
Abstract
Vision-language models map images and text into a joint embedding space. However, these embeddings often entangle multiple semantic features, which limits their interpretability and controllability. While sparse autoencoders have emerged as a useful tool for decomposing these embeddings into monosemantic features, their application to joint embedding spaces has largely relied on an implicit, untested assumption that semantically corresponding features share the same directions across modalities. In this paper, we challenge this assumption by identifying discrepancies in feature directions for the same concept across image and text modalities, a phenomenon we term cross-modal feature heterogeneity. We demonstrate that this heterogeneity is a key driver of the modality split, where a shared concept activates different latents depending on the modality. This finding further reveals why aligning latent activations alone is insufficient to resolve the underlying feature mismatch. Motivated by this observation, we propose an approach that trains modality-specific sparse autoencoders to preserve each modality's feature geometry, and then aligns corresponding features post hoc. Our method improves reconstruction fidelity and enhances performance in cross-modal retrieval and concept steering.