Recall Before Rerank: Benchmarking Deep Learning Models for Large-Scale Code-to-Code Retrieval
Benchmarking 17 deep models for large-scale code retrieval; introducing LLM-based normalization and rewriting improves recall by up to 29%.
Key Findings
Methodology
The study conducts large-scale empirical evaluation of 17 models across four datasets and five programming languages, measuring Precision@k and NDCG. Experiments involve over 920 runs on GPU clusters, comparing architectures like bi-encoders, encoder-decoder, and decoder-only LLMs. The research introduces LLM-based code normalization and query rewriting, analyzing their impact on recall. It assesses computational throughput and resource consumption, establishing trade-offs between efficiency and accuracy, with detailed ablation studies to validate strategies.
Key Results
- Models like Qwen3 Embedding and CodeXEmbed outperform others with Precision@50 exceeding 90% across datasets, but at high computational costs. Lightweight models such as StarEncoder achieve over 100 KB/s throughput but drop to 30% precision on complex tasks. Incorporating LLM-based normalization boosts recall of weaker models by up to 29%. The results highlight the necessity of task-specific model selection balancing speed and accuracy, with multi-stage retrieval architectures proving most effective.
- In large datasets (e.g., 6 million methods in BigCloneBench), lightweight models process tens of KB per second, suitable for SMEs, but underperform in semantic complexity. High-accuracy models, while precise, demand extensive GPU resources, making deployment challenging. Cross-dataset comparisons reveal significant performance gaps, emphasizing the importance of model scalability and resource considerations in real-world applications.
- Applying code rewriting strategies (variable renaming, comment insertion/deletion) enhances model robustness, especially for style-sensitive models like Code Llama, with improvements up to 29%. Top models like Qwen3 remain stable under style variations, validating hybrid approaches. The study also finds model size correlates with inference speed, necessitating careful architecture choices to meet practical constraints while maintaining high recall.
Significance
This work provides a comprehensive benchmark of deep learning models for large-scale code retrieval, addressing critical challenges in accuracy, efficiency, and scalability. It informs both academia and industry on optimal model selection, guiding the development of practical, high-performance code search systems. The introduction of LLM-based normalization and rewriting strategies offers new avenues to enhance recall, especially for resource-constrained environments. Overall, the findings advance the state-of-the-art in semantic code search, supporting smarter software engineering workflows and automated maintenance.
Technical Contribution
The paper establishes a systematic benchmark of 17 diverse models, including recent architectures like Qwen3 and CodeXEmbed, across multiple datasets and languages. It innovates by integrating LLM-based code normalization and query rewriting, significantly improving recall for weaker models. The study analyzes the relationship between model size, throughput, and accuracy, providing insights into scalable deployment. It proposes a hybrid two-stage retrieval framework, combining fast encoders with powerful rerankers, to optimize efficiency and precision, offering a practical blueprint for real-world systems.
Novelty
This is the first large-scale, multi-language, multi-dataset benchmark evaluating a broad spectrum of deep models for code retrieval, including the latest LLM-based approaches. The integration of LLM-driven code normalization and query rewriting as a means to boost recall is novel. Unlike prior work limited to small datasets or single architectures, this study provides a comprehensive, empirical foundation for model selection and system design in real-world, large-scale environments, bridging the gap between research and deployment.
Limitations
- Despite extensive evaluation, the models' performance drops significantly on complex semantic tasks and style variations, indicating robustness issues. High computational costs limit practical deployment, especially for large datasets. The experiments focus on specific datasets and languages, raising questions about generalization to unseen scenarios. Future work should explore model compression, adaptive rewriting, and broader language coverage to address these limitations.
Future Work
Future research will focus on model compression and acceleration techniques to reduce inference costs. Developing adaptive rewriting strategies that dynamically optimize for style and semantic variations is also a priority. Extending evaluations to more languages and real-world codebases will improve generalization. Integrating multi-modal data, such as documentation and execution traces, could further enhance retrieval robustness. Ultimately, the goal is to create scalable, accurate, and resource-efficient code search systems suitable for industry-scale deployment.
AI Executive Summary
The exponential growth of software repositories has made code retrieval an essential component of modern software engineering. Traditional keyword-based methods, while fast, lack semantic understanding, limiting their effectiveness. Deep learning models, especially bi-encoders and large language models, have significantly advanced the field by capturing code semantics more effectively. However, their high computational costs and scalability issues hinder widespread adoption in large-scale environments.
This study conducts a comprehensive benchmark of 17 models across multiple datasets and programming languages, evaluating their effectiveness and efficiency. The results reveal a stark trade-off: high-accuracy models like Qwen3 Embedding deliver near-perfect recall but require extensive resources, making them impractical for massive codebases. Conversely, lightweight models process code rapidly but suffer from poor recall in complex scenarios. To bridge this gap, the authors introduce LLM-based code normalization and query rewriting strategies, which improve recall by up to 29% for weaker models, demonstrating a promising hybrid approach.
The experiments also analyze the relationship between model size, throughput, and accuracy, emphasizing that deployment must consider task-specific requirements and hardware constraints. The findings advocate for a multi-stage retrieval architecture—fast, lightweight candidate generation followed by deep reranking—to balance speed and precision effectively.
Overall, this work provides a vital empirical foundation for designing scalable, efficient code retrieval systems. It highlights the importance of tailored model selection, the potential of LLM-based normalization, and the need for ongoing research into resource-efficient architectures. These insights are crucial for advancing automated code understanding, maintenance, and security in industry and academia alike.
Deep Dive
Abstract
Semantic code search and clone detection are essential for software development, maintenance, and reuse. This paper evaluates the effectiveness, efficiency, and scalability of contemporary deep learning models for first-stage recall in large-scale code-to-code search engines. Benchmarking across multiple programming languages and datasets reveals critical limits in the precision and scalability of these models on Terabyte-scale source-code collections. We present LLM-based code normalisation and query-rewriting schemes that yield significant gains in precision for lower-performing models. Our results question the sustainability of resource-constrained deployment and the assumed robustness of current code-specialised LLMs across datasets. We conclude with actionable insights for building scalable, efficient code-retrieval systems.