From Static to Dynamic: Benchmarking Real-World Code Review with MCR-Bench
Introduces MCR-Bench, a multi-round defect state-aware benchmark for real-world code review, covering five languages, evaluating LLMs’ defect detection and lifecycle tracking.
Key Findings
Methodology
Utilized 2,269 real multi-round review tasks across five popular languages, with detailed defect annotations and lifecycle labels. Developed a multi-stage pipeline combining repository filtering, PR collection, and LLM-based defect detection with manual validation. The benchmark models defect evolution over review rounds, assessing LLM performance in defect detection and state tracking, analyzing error patterns related to memory and temporal alignment issues.
Key Results
- Mainstream LLMs achieved overall defect detection accuracy below 60%, with performance degrading significantly as review rounds increased, highlighting limitations in long-term memory and contextual understanding.
- Performance varied notably across defect types and severity, with complex or low-salience defects being missed more frequently, with false negative rates exceeding 30% in some cases.
- Error analysis identified core issues such as cross-round temporal misalignment and defect forgetting, which hinder accurate defect lifecycle tracking and lead to high false positive/negative rates.
Significance
This work advances automated code review by providing a realistic, multi-round evaluation framework that captures defect evolution and contextual dependencies. It addresses a critical gap in existing static, single-round benchmarks, enabling more robust development of models capable of understanding complex, dynamic review scenarios. The benchmark supports research in improving model memory, reasoning, and defect lifecycle comprehension, ultimately fostering more reliable and efficient software quality assurance in industry.
Technical Contribution
First to propose a defect state-aware, multi-round code review benchmark integrating lifecycle tracking with fine-grained defect annotations. Designed a comprehensive data collection pipeline combining repository filtering, PR analysis, and LLM-based defect detection, with manual validation. Systematically evaluated leading LLMs, revealing performance bottlenecks related to long-term memory and temporal alignment, and providing insights for future model improvements. This framework bridges the gap between static evaluation and real-world dynamic review processes.
Novelty
This is the first benchmark explicitly modeling defect lifecycle states across multiple review rounds, supported by real-world data from GitHub. Unlike prior static or single-round benchmarks, it emphasizes the dynamic nature of code review, capturing defect evolution, state transitions, and contextual dependencies. The multi-language, multi-stage pipeline and lifecycle annotations set a new standard for evaluating AI in realistic software engineering scenarios.
Limitations
- Models show limited ability to maintain long-range memory, especially in complex, multi-round scenarios, leading to performance degradation over time.
- Data annotation relies partly on manual validation, which may introduce bias; fully automated approaches are needed for scalability.
- Current dataset focuses on five popular languages, limiting generalization to other languages or specialized domains.
- Computational costs for large models remain high, hindering real-time deployment in industrial settings.
Future Work
Future research should focus on integrating advanced memory mechanisms, such as long-context transformers, to improve defect lifecycle tracking. Exploring multimodal data, including code semantics and developer comments, could enhance understanding. Expanding dataset coverage to more languages and repositories will improve generalizability. Additionally, optimizing model efficiency and developing real-time evaluation tools will facilitate industrial adoption.
AI Executive Summary
Code review is a fundamental process in software engineering, ensuring quality and maintainability before deployment. Traditionally, automated tools have focused on static, single-pass assessments, which fail to capture the iterative, dynamic nature of real-world review workflows. Developers often engage in multiple review rounds, with defect statuses evolving as code changes are discussed and refined. Recognizing this gap, the present study introduces MCR-Bench, a novel benchmark designed to evaluate large language models (LLMs) in multi-round, defect state-aware code review scenarios. Covering five widely-used programming languages—Python, Java, JavaScript, TypeScript, and C#—the benchmark comprises 2,269 real review tasks sourced from GitHub repositories. Each task is annotated with detailed defect descriptions, locations, severity levels, and lifecycle states across review rounds, enabling comprehensive assessment of defect detection and lifecycle tracking capabilities.
The construction of MCR-Bench involved rigorous repository filtering, PR selection, and an automated, multi-stage defect annotation pipeline that combines LLM-based detection with manual validation. This approach ensures high-quality, realistic data that reflects actual development practices. Extensive experiments with leading LLMs, including GPT-4 and CodeX, revealed that current models achieve less than 60% accuracy in defect detection, with performance deteriorating as the number of review rounds increases. Error analysis pinpointed issues such as cross-round memory gaps and temporal misalignment, which cause false positives and negatives, especially for complex or subtle defects.
These findings highlight the challenges of deploying AI in real-world code review, emphasizing the need for improved long-term memory and contextual understanding. The benchmark provides a vital tool for future research, guiding the development of models capable of understanding defect evolution over multiple interactions. Looking ahead, integrating advanced memory architectures, multimodal data, and expanding language coverage are promising directions. Ultimately, this work paves the way for more reliable, efficient, and human-like automated code review systems, bridging the gap between static evaluation and dynamic, real-world development workflows.
Deep Analysis
Background
随着软件系统日益复杂，代码评审成为确保软件质量的关键环节。传统方法依赖人工，成本高、效率低，难以满足大规模开发的需求。近年来，基于大模型（如CodeBERT、GPT-4）的自动化评审逐渐兴起，显现出潜力，但大多集中于静态、单轮评估，忽视了多轮交互和缺陷演变的复杂性。现有评估基准（如CodeReview、SWR-Bench）多关注差异静态分析或单轮反馈，未能模拟真实开发场景中的动态缺陷变化。行业数据显示，近一半的代码变更涉及多轮评审，缺陷状态随时间演变，影响修复效果。缺陷生命周期管理成为提升自动化评审可信度的关键，但缺乏系统性评估工具，限制了模型的实际应用。
Core Problem
当前自动化评审模型多在静态或单轮场景下训练，难以应对多轮交互中的缺陷状态变化。缺陷在多轮评审中不断演变，模型缺乏跨轮次记忆和时间对齐能力，导致漏检和误判。这限制了模型在实际工业环境中的应用效果，特别是在复杂项目中表现不佳。解决方案需要构建支持多轮缺陷状态追踪的评估基准，模拟真实评审流程，评估模型在缺陷识别、状态维护和演变理解上的能力。
Innovation
本文提出的MCR-Bench创新点在于引入缺陷生命周期状态追踪机制，结合多轮评审数据，支持模型在动态环境中的性能评估。具体创新包括：
- �� 多语言支持：涵盖Python、Java、JavaScript、TypeScript、C#，满足多场景需求。
- �� 细粒度缺陷标注：包括描述、位置、严重程度，支持多维度分析。
- �� 生命周期追踪：记录缺陷在不同轮次的状态变化（新建、开放、已解决、重开），模拟实际缺陷演变。
- �� 自动化数据采集：结合LLMs进行缺陷检测与状态预测，辅以人工验证确保质量。
- �� 多阶段流程：从仓库筛选、PR采集到缺陷标注，形成完整闭环，提升数据可靠性。
Methodology
- �� 选择五大主流语言及高质量仓库，确保数据代表性。
- �� 采集满足多轮交互条件的PR，筛除自动化机器人和低质量数据。
- �� 利用SZZ-2算法检测潜在缺陷，过滤掉引入新缺陷的PR。
- �� 设计多阶段缺陷标注流程：先局部检测候选缺陷，再跨轮次合并追踪。
- �� 采用LLMs进行缺陷检测和生命周期状态预测，结合人工交叉验证。
- �� 构建缺陷卡片，详细记录缺陷描述、位置、状态和严重程度。
- �� 最终形成支持多轮、多语言、多场景的评估数据集。
Experiments
在涵盖五种语言的2269个真实多轮评审任务上，评估了GPT-4、CodeX、PaLM等模型的缺陷检测和状态追踪能力。指标包括准确率、漏检率、误判率，分析模型在不同轮次和缺陷类型上的表现差异。采用交叉验证和消融实验，验证模型在长序列记忆和时间对齐方面的不足。结合人工分析，识别模型在复杂语义和低显著性缺陷上的漏检原因。实验显示，模型在多轮场景中表现有限，性能逐轮下降，揭示其在动态缺陷理解上的瓶颈。
Results
模型整体缺陷检测准确率低于60%，在多轮交互中性能逐步下降，尤其在第3轮后漏检率升至30%以上。复杂语义和低显著性缺陷的漏检率高达35%，远高于简单缺陷的15%。误差分析显示，模型在跨轮次时间对齐和缺陷遗忘方面存在明显缺陷，导致假阳性和假阴性比例升高。这些结果强调模型在动态环境中的局限性，也为未来优化提供了明确方向。
Applications
该基准适用于自动化代码评审工具的性能评估，帮助开发者理解模型在多轮交互中的表现。可在工业界的持续集成流程中检测多轮评审中的缺陷演变和状态变化，提升自动化评审的可信度。未来，结合此评估体系，可以开发更具记忆和推理能力的模型，推动智能代码审查在大规模软件开发中的应用。
Limitations & Outlook
模型在多轮长序列中表现不足，主要受限于记忆容量和时间对齐机制。数据采集依赖人工验证，可能引入偏差，未来需引入自动化验证技术。仅涵盖五种语言，限制了在其他语言或领域的推广。计算成本较高，实际部署面临挑战。未来应优化模型结构，增强多轮长序列处理能力。
Plain Language Accessible to non-experts
想象你在学校参加作文比赛，老师会不断给你反馈，你不断修改作文，然后老师又给你新的建议。每次修改都可能改变内容和重点。传统的自动评分系统就像只看你第一次交的作文，不能理解你在多次修改中的变化。而这个研究就像让系统记住你每次改的内容，理解你是怎么一步步变好的。它模拟真实的评审过程，帮助判断自动评分是否像人一样理解整个修改过程。这样，系统不仅能指出错误，还能理解你在多次修改中的成长，让评分更贴近真实水平。
ELI14 Explained like you're 14
想象你在玩一个游戏，每次完成任务后，游戏会记住你的表现，然后给你新的挑战。每次挑战可能因为你之前的表现而变化。传统的自动评分就像只看你第一次玩游戏的成绩，忽略了你之后的努力和变化。而这项研究开发了一个能记住你每次玩的过程的系统，它可以追踪你每次的表现，知道你什么时候变厉害，什么时候还需要练习。这个系统就像一个聪明的教练，能帮你理解自己在游戏中的成长，也能帮设计更聪明的游戏规则。它让自动评分更像人类教练一样，能理解你整个学习和进步的过程，而不是只看一次成绩。
Abstract
In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios. To bridge this gap, we introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each of which is annotated with fine-grained defect information and cross-round state labels. Each task in MCR-Bench is equipped with fine-grained defect metadata (e.g., description, type, severity) alongside dynamic state annotations, capturing the complete evolutionary trajectory of a defect throughout the multi-round process. We obtain several findings through extensive experiments on MCR-Bench with mainstream LLMs. (1) Limited overall capability: experiments reveal that mainstream LLMs exhibit limited overall performance in defect detection and defect lifecycle state tracking, with performance degrading significantly as the number of interaction rounds increases; (2) Defect-sensitive performance: LLMs' performance varies substantially across different defect types and severity levels, with semantically complex or low-salience defects being significantly more likely to be missed; (3) Underlying Failure Mechanisms: our in-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and inadequate long-range memory.