Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Banks
Proposes Dual Bank Normalization (DBNORM) with DualIS and DualDIS to mitigate hubness, boosting cross-modal retrieval performance.
Key Findings
Methodology
This work provides a theoretical analysis demonstrating that hubs exhibit high similarity across modalities, emphasizing the importance of utilizing both query and gallery data. Based on this, the authors introduce DBNORM, which constructs query and gallery banks from training data, applying DualIS and DualDIS normalization techniques. These methods dynamically adjust similarity distributions by inverting softmax probabilities, reducing hub dominance. The process involves bank construction, similarity computation, normalization, and ranking, effectively alleviating hubness in multi-modal retrieval tasks.
Key Results
- On eight benchmarks including MSR-VTT, MSVD, and ActivityNet, DBNORM outperforms traditional methods like IS and DIS, improving R@1 by approximately 3-4 percentage points. For instance, in MSR-VTT full split, DualDIS raises R@1 from 44.10 to 45.00, surpassing previous post-processing approaches. In text-image retrieval on MSCOCO and Flickr30k, R@1 increases by about 3-4 points, confirming generalization. Ablation studies show that precomputing activation sets enhances robustness, especially under data bias or imperfect banks.
- Theoretical analysis confirms that hubs exhibit high similarity across modalities, justifying the use of both query and gallery banks. Empirical results demonstrate that combining these banks effectively reduces skewness, leading to more balanced retrieval outcomes. The methods maintain computational efficiency and scalability, suitable for large datasets.
Significance
This research addresses a fundamental challenge in multi-modal retrieval—hubness—by providing a theoretically grounded, practical post-processing solution. It bridges the gap between theoretical understanding of hub behavior and real-world retrieval performance, enabling more fair and accurate cross-modal matching. The approach enhances robustness and scalability, with broad implications for multimedia content search, recommendation systems, and AI-driven information retrieval, fostering advancements in both academia and industry.
Technical Contribution
The main innovations include the formulation of a theoretical framework proving hubs’ high similarity across modalities, and the development of DBNORM, which integrates query and gallery banks for adaptive similarity normalization. DualIS and DualDIS algorithms extend softmax inversion techniques, with DualDIS leveraging precomputed activation sets for robustness. These contributions differ from prior methods by explicitly exploiting cross-modal bank information, offering improved theoretical guarantees and practical performance in mitigating hubness.
Novelty
This is the first comprehensive study to incorporate both query and gallery banks into a unified normalization framework for cross-modal retrieval. The dual softmax-based algorithms, combined with theoretical validation, represent a significant step forward in addressing hubness. Unlike previous approaches that relied solely on query data or static normalization, this work dynamically adjusts similarity scores using cross-modal bank information, setting a new standard for post-processing in multi-modal retrieval.
Limitations
- The effectiveness depends on the quality and representativeness of the training banks; biased or limited samples can impair performance, especially in zero-shot or few-shot scenarios.
- Precomputing activation sets introduces additional storage and computational overhead, which may challenge scalability in extremely large datasets.
- The current approach assumes static banks; dynamic or evolving data distributions require further adaptation, and online updating mechanisms are yet to be developed.
Future Work
Future research will explore adaptive bank updating strategies, integrating online learning to handle dynamic data environments. Combining this framework with advanced feature fusion techniques may further improve retrieval accuracy. Extending the approach to unsupervised or zero-shot settings could broaden applicability. Additionally, optimizing computational efficiency for large-scale deployment remains a key direction.
AI Executive Summary
Cross-modal retrieval has become a cornerstone of multimedia information access, enabling users to search across images, videos, audio, and text seamlessly. Despite significant advances driven by deep neural networks like CLIP, ALBEF, and X-Pool, a persistent challenge remains—hubness. This phenomenon, where a small subset of gallery points repeatedly appear as nearest neighbors, skews retrieval results, diminishes fairness, and hampers overall performance.
Traditional solutions focus on training strategies or static post-processing methods such as CSLS or GC, but these often fall short in addressing the fundamental causes of hubness. Recognizing this, the authors propose a novel post-processing framework—Dual Bank Normalization (DBNORM)—which leverages both query and gallery banks constructed from training data. This approach is grounded in a theoretical analysis demonstrating that hubs exhibit high similarity across different modalities, making their identification and mitigation feasible through cross-modal information.
The core of DBNORM involves two innovative normalization techniques: Dual Inverted Softmax (DualIS) and Dual Dynamic Inverted Softmax (DualDIS). DualIS inverts the softmax probabilities, effectively reducing the influence of hubs, while DualDIS precomputes activation sets to dynamically refine similarity adjustments. These methods are computationally efficient, scalable, and adaptable, leading to substantial improvements across diverse benchmarks such as MSR-VTT, MSVD, COCO, and Flickr30k. Results show consistent gains in R@1, often exceeding 3 percentage points over prior state-of-the-art post-processing methods.
The significance of this work lies in its theoretical rigor and practical effectiveness. By explicitly modeling the high similarity of hubs across modalities, it offers a principled solution to a long-standing problem. The approach enhances the fairness and robustness of cross-modal retrieval systems, with promising applications in multimedia search, recommendation engines, and AI-driven content understanding. Future directions include adaptive bank updating, online learning integration, and extending to unsupervised scenarios, paving the way for more intelligent and scalable multi-modal AI systems.
Deep Dive
Abstract
In this work, we present a post-processing solution to address the hubness problem in cross-modal retrieval, a phenomenon where a small number of gallery data points are frequently retrieved, resulting in a decline in retrieval performance. We first theoretically demonstrate the necessity of incorporating both the gallery and query data for addressing hubness as hubs always exhibit high similarity with gallery and query data. Second, building on our theoretical results, we propose a novel framework, Dual Bank Normalization (DBNorm). While previous work has attempted to alleviate hubness by only utilizing the query samples, DBNorm leverages two banks constructed from the query and gallery samples to reduce the occurrence of hubs during inference. Next, to complement DBNorm, we introduce two novel methods, dual inverted softmax and dual dynamic inverted softmax, for normalizing similarity based on the two banks. Specifically, our proposed methods reduce the similarity between hubs and queries while improving the similarity between non-hubs and queries. Finally, we present extensive experimental results on diverse language-grounded benchmarks, including text-image, text-video, and text-audio, demonstrating the superior performance of our approaches compared to previous methods in addressing hubness and boosting retrieval performance. Our code is available at https://github.com/yimuwangcs/Better_Cross_Modal_Retrieval.