M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
Constructed M2RC-EVAL, a multilingual repository-level code completion benchmark with AST-based fine-grained annotations for 18 languages.
Key Findings
Methodology
This work develops M2RC-EVAL, a comprehensive benchmark covering 18 languages, utilizing Tree-sitter to parse source code into ASTs. Nodes are annotated with 10-layer bucket labels and 11 semantic categories, enabling detailed performance analysis. Combined with the M2RC-INSTRUCT multilingual instruction corpus, the framework enhances model training. Evaluation metrics include exact match (EM) and edit similarity (ES), assessing models across diverse code structures and semantics, demonstrating significant improvements after fine-tuning and retrieval augmentation.
Key Results
- Models fine-tuned on M2RC-INSTRUCT achieved an average EM of 45.4%, a 30% increase over baseline. Retrieval strategies improved performance by over 20%, especially in cross-file contexts. DeepSeekCoder-6.7B excelled in Go (EM 54.4%), while weaker in HTML (EM 41%), underscoring the importance of multilingual evaluation.
- Fine-grained annotations revealed lower performance on shallow AST nodes like declarations and expressions, highlighting areas for future optimization. Model performance varied significantly across semantic levels, with deeper nodes showing higher accuracy.
- The approach effectively bridges the gap between structural code understanding and semantic comprehension, providing a robust foundation for future multi-task, multi-modal code intelligence systems.
Significance
This research addresses the critical need for a unified, multi-language benchmark for repository-level code completion, enabling precise evaluation of models in real-world scenarios. By integrating AST-based structural and semantic annotations, it offers a multi-dimensional view of model capabilities, fostering advances in automated programming, intelligent code assistants, and software engineering productivity. The benchmark's comprehensive design supports industry adoption, guiding the development of more accurate, generalizable AI coding tools, thus pushing the frontier of AI-assisted software development.
Technical Contribution
The key innovations include: 1) constructing a large-scale, multi-language repository benchmark with 18 languages; 2) leveraging Tree-sitter to generate multi-layer AST annotations; 3) defining 11 semantic categories for fine-grained labeling; 4) integrating instruction fine-tuning with retrieval strategies to enhance model performance. These contributions enable detailed performance diagnostics and facilitate targeted improvements in code LLMs, representing a significant step beyond prior single-language or coarse evaluation frameworks.
Novelty
This study pioneers the combination of AST-based hierarchical and semantic annotations across a broad set of 18 languages for repository-level code completion. Unlike previous benchmarks limited to few languages or superficial metrics, it introduces a multi-dimensional evaluation framework that captures structural, semantic, and contextual performance, providing a new standard for comprehensive assessment of multilingual code intelligence.
Limitations
- The AST-based annotations depend on parser accuracy, which may introduce errors in complex or ambiguous syntax structures, affecting evaluation reliability.
- Metrics focus on surface-level similarity (EM, ES), insufficient for measuring true semantic understanding and correctness.
- Model tuning relies on static datasets, lacking dynamic context adaptation, which limits real-world applicability where code evolves rapidly.
Future Work
Future directions include integrating dynamic code execution feedback, expanding semantic categories, and developing semantic-aware evaluation metrics. Additionally, exploring multi-modal data (e.g., runtime performance, documentation) and reinforcement learning techniques could further improve model robustness and generalization in real-world software engineering tasks.
AI Executive Summary
In the rapidly evolving field of automated software engineering, the ability of large language models (LLMs) to perform multilingual, repository-level code completion remains underexplored. Existing benchmarks are limited to a handful of languages and lack fine-grained analysis of structural and semantic performance. To address this gap, this study introduces M2RC-EVAL, a comprehensive benchmark covering 18 major programming languages, leveraging AST parsing to generate multi-layered structural annotations and semantic labels. This approach enables detailed performance diagnostics across diverse code constructs, from declarations to complex control flows.
The core innovation lies in combining AST-based hierarchical labels with semantic categories, providing a multi-dimensional evaluation framework. The benchmark is complemented by M2RC-INSTRUCT, a multilingual instruction corpus designed to fine-tune models for better cross-language generalization. Experimental results demonstrate that models fine-tuned on this corpus, especially with retrieval augmentation, achieve significant performance gains, with EM scores reaching 45.4%, a substantial improvement over baseline models.
The significance of this work extends beyond benchmarking. It offers a scalable, detailed assessment platform that can guide future model development, emphasizing structural understanding and semantic comprehension. The methodology and datasets set a new standard for evaluating multilingual code intelligence, fostering progress in automated programming, intelligent code assistants, and software engineering productivity.
Despite these advances, challenges remain. AST annotations depend on parser accuracy, and current metrics do not fully capture semantic correctness. Future work will focus on dynamic evaluation, broader semantic categories, and multi-modal data integration, aiming to develop more robust, context-aware AI coding systems that can adapt to the complexities of real-world software development.
Deep Dive
Abstract
Repository-level code completion has drawn great attention in software engineering, and several benchmark datasets have been introduced. However, existing repository-level code completion benchmarks usually focus on a limited number of languages (<5), which cannot evaluate the general code intelligence abilities across different languages for existing code Large Language Models (LLMs). Besides, the existing benchmarks usually report overall average scores of different languages, where the fine-grained abilities in different completion scenarios are ignored. Therefore, to facilitate the research of code LLMs in multilingual scenarios, we propose a massively multilingual repository-level code completion benchmark covering 18 programming languages (called M2RC-EVAL), and two types of fine-grained annotations (i.e., bucket-level and semantic-level) on different completion scenarios are provided, where we obtain these annotations based on the parsed abstract syntax tree. Moreover, we also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code LLMs. Comprehensive experimental results demonstrate the effectiveness of our M2RC-EVAL and M2RC-INSTRUCT.