CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models
Proposes CodeCriticBench, a comprehensive benchmark for large language models' code critique, combining basic and fine-grained multi-task evaluation with 4,300 samples.
Key Findings
Methodology
This study constructs a holistic benchmark with 4,300 samples covering code generation and question-answering tasks. It employs multi-level evaluation, including correctness classification and detailed multi-dimensional scoring, integrating expert and model assessments. Scores are calibrated via linear regression. Data collection combines algorithmic problem sets, StackOverflow, and automated generation, with difficulty levels assigned based on model correctness rates. Metrics include accuracy (ACC), mean squared error (MSE), and Pass@1, providing multi-angle evaluation of critique capabilities.
Key Results
- Across 38 models, larger parameter sizes correlate with higher critique accuracy, reaching up to 75.30%. Deep models excel in error detection and fine-grained scoring, with Claude-3.5-Sonnet achieving an MSE of 1.02 in code QA.
- Models perform well on easy samples (>90% accuracy) but drop to around 30% on hard samples, indicating room for improvement in complex scenarios.
- Larger models outperform smaller ones in error type recognition, especially for common errors like reference errors and performance issues, demonstrating scale-performance correlation.
Significance
This benchmark fills a critical gap by providing a multi-dimensional, fine-grained evaluation framework for code critique, advancing research in automated code review and intelligent debugging. It offers a standardized tool to quantify models’ understanding and analytical abilities, fostering development of more reliable and intelligent programming assistants. The systematic assessment reveals how model size influences critique performance, guiding future model design and optimization. Ultimately, it accelerates the deployment of robust AI-powered coding tools in industry, improving software quality and developer productivity.
Technical Contribution
This work introduces the first comprehensive multi-task, multi-dimensional code critique benchmark, integrating correctness and detailed scoring. It employs a calibration mechanism to improve score consistency, evaluates 38 models across scales, and establishes the positive relationship between model size and critique performance. The framework’s design allows scalable, reproducible, and extensible evaluation, setting a new standard for assessing AI in software engineering tasks.
Novelty
Unlike prior benchmarks focused solely on correctness or single tasks, CodeCriticBench offers a multi-task, multi-dimensional evaluation with expert and model scoring, using fine-grained checklists. It systematically measures large models’ critique capabilities, representing a significant step forward in comprehensive AI evaluation for code understanding and analysis.
Limitations
- The evaluation relies mainly on static code samples, lacking dynamic execution-based error detection and repair capabilities.
- Calibration via linear regression may introduce biases in extreme cases or rare error types.
- Current metrics do not fully capture real-world coding scenarios, such as integration with development environments or multi-modal data, necessitating future extensions.
Future Work
Future directions include integrating dynamic execution data, expanding multi-modal assessments, and developing reinforcement learning-based critique strategies. Additionally, broadening the benchmark to cover more programming languages and real-world development workflows will enhance its applicability, ultimately fostering more capable and trustworthy AI coding assistants.
AI Executive Summary
As large language models (LLMs) continue to revolutionize code understanding and generation, evaluating their critique abilities becomes increasingly vital. Existing benchmarks predominantly focus on reasoning tasks, with limited emphasis on code-specific analysis, especially in multi-dimensional and fine-grained contexts. This gap hampers progress in deploying models for critical software engineering tasks such as automated code review, debugging, and optimization.
To address this, we introduce CodeCriticBench, a comprehensive benchmark designed to evaluate LLMs’ critique capabilities across code generation and question-answering tasks. The benchmark comprises 4,300 samples, spanning various difficulty levels and scenarios, with meticulous data collection from algorithmic problem sets, StackOverflow, and synthetic generation. Each sample is annotated with correctness labels, multi-dimensional evaluation checklists, and scores calibrated through linear regression, ensuring robust and objective assessment.
Our evaluation of 38 models reveals a clear correlation between model size and critique performance, with the largest models achieving up to 75.30% accuracy. Deep models excel in error detection and detailed scoring, particularly in identifying common errors like reference and performance issues. Notably, models perform well on easy samples but face challenges with complex, hard scenarios, highlighting areas for future improvement.
This work establishes a new standard for multi-task, multi-dimensional code evaluation, providing valuable insights for both academia and industry. It paves the way for developing more intelligent, reliable AI-powered coding tools, ultimately transforming software development workflows. Future efforts will focus on integrating dynamic execution data, expanding language coverage, and refining evaluation metrics to better mirror real-world coding environments, fostering the next generation of AI-assisted programming systems.
Deep Analysis
Background
近年来，随着Transformer架构的广泛应用，LLMs在自然语言处理和代码生成领域取得了突破性进展。OpenAI的GPT系列、Meta的LLaMA、以及专门针对代码的CodeLlama、StarCoder等模型，展现出强大的理解和生成能力。早期工作如Humaneval和MBPP提供了基础的代码正确性测试集，但缺乏对模型批评能力的系统评估。CriticBench等基准开始关注模型的批评能力，但多局限于单一任务或维度，未能全面衡量模型在复杂场景中的表现。随着模型参数的不断扩大，评估其深度理解、错误诊断和建议能力成为新挑战。现有研究多强调生成质量，少有系统性衡量模型在代码审查、错误检测、改进建议等方面的能力，限制了模型在实际软件开发中的应用潜力。
Core Problem
当前大模型在代码理解与批评方面仍存在不足，主要表现为：缺乏多维度、细粒度的评估体系，难以全面衡量模型的深度分析和改进建议能力；现有基准多偏重于正确性检测，忽略模型的解释和建议能力；此外，缺少涵盖不同任务、难度和场景的统一评测框架。这些问题限制了模型在自动代码审查、智能辅导等实际场景中的应用效果，也阻碍了模型优化的方向。解决这些瓶颈，亟需构建一个多任务、多维度、细粒度的评测体系，系统性衡量模型的批评能力，为模型改进提供科学依据。
Innovation
本研究的核心创新在于：
- �� 提出涵盖代码生成与问答的全方位批评基准，结合基础正确性和细粒度多维评分体系。
- �� 设计多层次评估策略，包括模型的基本正确性判定和细粒度的多维评分，结合专家与模型的评分，确保评估的全面性与客观性。
- �� 引入模型校准机制，通过线性回归校正模型评分，提升评估结果的稳定性与一致性。
- �� 大规模评测38个不同参数规模的模型，验证模型规模与批评能力的正相关关系，为模型设计提供理论支持。
Methodology
- �� 数据采集：结合算法题库（CodeForces、MBPP、LiveCodeBench）和StackOverflow，自动生成与筛选样本，确保多样性与质量。
- �� 样本难度划分：利用12个先进模型的正确率，将样本划分为Easy（>80%正确）、Medium（60-80%）、Hard（<60%）三类。
- �� 任务设计：涵盖代码生成（带测试用例验证正确性）与代码问答（由人工评估正确性）两个任务。
- �� 评估指标：采用准确率（ACC）、均方误差（MSE）和Pass@1，结合专家评分进行校准。
- �� 细粒度评分：定义10个维度，通过模型与专家评估，结合线性回归校正，获得最终评分。
Experiments
- �� 评测38个不同参数规模的模型，包括开源和闭源模型。
- �� 采用多任务、多难度、多场景的样本，确保评估的全面性。
- �� 指标包括基础的正确性识别、错误类型分类、细粒度多维评分。
- �� 通过对比不同模型参数规模、任务难度和场景，分析模型性能变化趋势。
- �� 进行消融实验验证评分体系的有效性，分析模型规模与性能的关系。
Results
- �� 模型参数越大，批评准确率越高，最大达75.30%；深度模型在错误识别和细粒度评分中表现优异。
- �� 在不同难度样本中，易样本ACC超过90%，困难样本平均仅30%，显示模型在复杂场景中的不足。
- �� 大模型在错误类型识别上表现更佳，尤其在引用错误和性能问题方面，参数越多，准确率越高。
Applications
- �� 自动化代码审查：模型可作为辅助工具，识别潜在错误并提供改进建议。
- �� 智能编程辅导：为学习者提供详细的代码分析和优化建议，提升学习效率。
Limitations & Outlook
- �� 评测主要基于静态样本，未充分考虑动态执行中的错误检测。
- �� 校准机制依赖线性回归，可能在极端样本下偏差较大。
- �� 未来需引入多模态数据和实际场景测试，提升评估的实用性与全面性。
Plain Language Accessible to non-experts
想象一个工厂里有很多工人，他们每天都在制造各种产品。有些工人非常擅长发现自己做错了什么，并能提出改进方法。这个研究就像是在评估这些工人——也就是大模型——他们能不能像工厂里的工人一样，自己检查自己的工作，找到错误，并提出改正建议。研究设计了一个“测试场”，让模型在不同难度的任务中表现，像是让工人面对简单的产品和复杂的机械问题。通过观察他们的表现，就能知道这些模型是不是能像优秀的工人一样，帮忙检查和改进代码。这个“测试场”还会用不同的评分标准，像是工厂里的质量检查，让我们更全面了解模型的能力。最终，这个方法能帮助我们打造更聪明、更可靠的自动化代码审查工具，就像培训工人一样，让他们变得更厉害。
ELI14 Explained like you're 14
想象你在学校里，有一群同学在写作文。有的作文很棒，但有些会出现拼写错误、逻辑不清或者句子不通顺。老师想知道这些同学的作文哪里写得好，哪里需要改进。这个研究就像是设计了一个特别的“作文检查员”，它可以自己找出作文中的错误，还能给出改进建议。这个“检查员”经过训练，能在不同难度的作文中表现得很好，比如简单的作文和复杂的长篇大论。它不仅能告诉你“对”或“错”，还能详细说明为什么错，哪里可以写得更好。这样一来，学生们就可以根据这些建议不断提高写作水平。这个“作文检查员”就像是一个超级聪明的老师助手，帮助大家写出更棒的作文！
Abstract
The critique capacity of Large Language Models (LLMs) is essential for reasoning abilities, which can provide necessary suggestions (e.g., detailed analysis and constructive feedback). Therefore, how to evaluate the critique capacity of LLMs has drawn great attention and several critique benchmarks have been proposed. However, existing critique benchmarks usually have the following limitations: (1). Focusing on diverse reasoning tasks in general domains and insufficient evaluation on code tasks (e.g., only covering code generation task), where the difficulty of queries is relatively easy (e.g., the code queries of CriticBench are from Humaneval and MBPP). (2). Lacking comprehensive evaluation from different dimensions. To address these limitations, we introduce a holistic code critique benchmark for LLMs called CodeCriticBench. Specifically, our CodeCriticBench includes two mainstream code tasks (i.e., code generation and code QA) with different difficulties. Besides, the evaluation protocols include basic critique evaluation and advanced critique evaluation for different characteristics, where fine-grained evaluation checklists are well-designed for advanced settings. Finally, we conduct extensive experimental results of existing LLMs, which show the effectiveness of CodeCriticBench.