CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation
CodeVisionary employs a two-stage multi-agent framework combining requirement-driven context distillation and collaborative scoring for complex code evaluation.
Key Findings
Methodology
This paper introduces CodeVisionary, an innovative multi-agent evaluation framework integrating requirement-guided multi-dimensional context distillation with negotiation-based collaborative scoring. The process begins with an LLM-based environment setup, decomposing task requirements into detailed sub-requirements, and systematically collecting multi-faceted contextual information—such as latest technology knowledge, visual interaction cues, runtime data, and static analysis results. During evaluation, multiple LLM judges analyze the code from diverse perspectives through self-guided and negotiation strategies, leveraging external tools like static analyzers, unit tests, and environment simulators. This collaborative process ensures comprehensive, fine-grained assessment, with the final output being a detailed evaluation report that explains strengths and weaknesses, enhancing interpretability and reliability.
Key Results
- On a benchmark of 363 samples across 37 coding scenarios and 23 programming languages, CodeVisionary outperformed three baselines, with average Pearson, Spearman, and Kendall-Tau coefficients improved by 0.217, 0.163, and 0.141 respectively, demonstrating superior capability in complex code evaluation.
- The framework maintained stable performance across diverse languages and scenarios, especially excelling in tasks involving recent technologies and multiple requirements, validating the effectiveness of multi-dimensional context integration.
- The multi-judge negotiation mechanism significantly enhanced evaluation granularity and interpretability, providing detailed reports that guide developers in code improvement.
Significance
This work advances the field of automated code evaluation by addressing the limitations of existing methods in complex, multi-requirement scenarios. By integrating multi-agent collaboration, CodeVisionary offers a more accurate, explainable, and adaptable evaluation system, which is crucial for deploying reliable AI-assisted coding tools in industry. Its ability to incorporate multi-source contextual information and produce detailed assessments paves the way for real-time, continuous quality monitoring in software development pipelines, ultimately fostering more robust and trustworthy AI-driven programming workflows.
Technical Contribution
Technically, CodeVisionary introduces a multi-agent collaboration paradigm, combining environment construction, requirement decomposition, plan formulation, and multi-turn analysis within a unified framework. It leverages advanced LLM prompting strategies and external static/dynamic analysis tools to ensure comprehensive evaluation. The negotiation-based scoring mechanism allows multiple judges to iteratively refine their assessments, improving accuracy and robustness. The framework’s modular design facilitates extension to new tools and scenarios, representing a significant step beyond traditional static prompt-based evaluation methods, and offering a scalable solution for complex code assessment.
Novelty
This study is the first to propose a multi-agent, requirement-guided evaluation framework tailored for complex code scenarios. Unlike prior approaches relying solely on static prompts or single-model judgments, CodeVisionary’s collaborative, multi-perspective assessment captures nuanced issues and provides detailed explanations. Its integration of environment simulation, multi-source contextual data collection, and negotiation-based scoring constitutes a novel paradigm, addressing critical gaps in current automated evaluation techniques for complex, multi-requirement code generation.
Limitations
- The framework's reliance on multiple external tools and multi-turn interactions results in high computational overhead, limiting real-time deployment at scale.
- Its effectiveness in extremely large codebases or highly complex, domain-specific scenarios remains to be validated, indicating potential scalability challenges.
- Despite multi-judge collaboration, subjective biases may still influence evaluations; further standardization and automation are needed to improve consistency.
Future Work
Future efforts will focus on optimizing computational efficiency, possibly through model distillation or parallel processing, to enable real-time evaluation. Expanding the benchmark to include more diverse, real-world datasets will test generalization. Integrating the framework into CI/CD pipelines for continuous, automated quality control is also planned. Additionally, enhancing the negotiation mechanism with reinforcement learning could further improve judgment consistency and robustness, paving the way for fully autonomous, scalable code evaluation systems.
AI Executive Summary
The rapid advancement of large language models (LLMs) like OpenAI Codex and GPT-4 has revolutionized code generation, enabling AI to produce increasingly complex and functional programs. However, evaluating the quality of such generated code remains a significant challenge, especially in complex scenarios involving multiple requirements, diverse technologies, and interactive components. Traditional evaluation methods—human review, static metrics, or single-model assessments—are either labor-intensive, limited in scope, or lack explainability. This gap hampers the deployment of reliable AI-assisted coding tools in real-world software development pipelines.
To address these limitations, this paper introduces CodeVisionary, a novel multi-agent evaluation framework that combines requirement-guided context distillation with collaborative scoring. The core idea is to emulate a team of evaluators—each with specialized roles—working together to analyze code from multiple perspectives. The process begins with environment construction, where the framework sets up a complete, executable programming environment using Docker. It then decomposes complex task requirements into manageable sub-tasks, systematically gathering multi-dimensional contextual information such as the latest technological updates, visual interaction cues, runtime data, and static analysis results.
During evaluation, multiple LLM-based judges analyze the code through iterative steps, invoking external tools like linters, unit tests, and web browsers to verify correctness, style, and compliance. These judges engage in self-guided reasoning and negotiation, iteratively refining their assessments to reach consensus. This collaborative process ensures a fine-grained, comprehensive evaluation that captures subtle issues often missed by single-model methods. The final output is a detailed report explaining the code’s strengths and weaknesses, providing actionable insights for developers.
Extensive experiments on a benchmark of 363 samples across 37 scenarios and 23 languages demonstrate that CodeVisionary significantly outperforms existing baselines, with average improvements of 0.217, 0.163, and 0.141 in Pearson, Spearman, and Kendall-Tau coefficients respectively. The results validate its robustness and adaptability across diverse, complex coding tasks. By integrating multi-source contextual understanding and multi-judge collaboration, this framework paves the way for more reliable, explainable, and scalable automated code evaluation systems.
Looking ahead, future work will focus on optimizing computational efficiency, expanding application scenarios, and integrating the framework into industrial CI/CD pipelines for real-time quality monitoring. Despite current limitations in scalability and resource demands, CodeVisionary marks a significant step toward intelligent, comprehensive evaluation of AI-generated code, promising to enhance the trustworthiness and utility of AI in software engineering.
Deep Dive
Glossary
Multi-agent system (多智能体系统)
由多个自主智能体协作完成复杂任务的系统，能模拟人类团队合作，提升任务处理的全面性和鲁棒性。In this paper, used to实现多角度评估。
描述多评审协作机制的核心技术基础。
上下文蒸馏 (Context distillation)
从多源信息中提取关键特征，形成简洁、全面的上下文表示。用于确保模型对复杂场景的理解。In this paper, 指逐步采集技术、视觉、运行信息。
用于描述需求理解和信息采集环节。
静态分析 (Static analysis)
在不执行代码的情况下检测潜在问题的技术，包括语法、风格和潜在缺陷分析。In this paper, 结合外部工具提升评估准确性。
作为评估流程的重要组成部分。
协商机制 (Negotiation mechanism)
多个评审智能体通过多轮交流达成一致的评分和评价。增强评估的细粒度和可信度。In this paper, 实现多评审的合作与共识。
描述多智能体合作的核心策略。
环境构建 (Environment construction)
通过容器等技术搭建完整、可执行的编程环境，确保代码测试的真实性。In this paper, 作为评估前的基础步骤。
确保评估的真实性和可复现性。
Open Questions Unanswered questions from this research
- 1 如何进一步降低多智能体协作的计算成本，提升大规模场景下的效率，是未来研究的重要方向。
- 2 多模态信息（如视觉、交互数据）在评估中的融合策略仍需优化，以应对更复杂的应用场景。
- 3 评审主观性和一致性问题，如何通过自动化和标准化机制进一步提升评估的客观性和稳定性，仍待探索。
Applications
Immediate Applications
自动化代码质量检测
将CodeVisionary集成到软件开发流程中，实时监控生成代码的质量，辅助开发者快速定位问题，提升开发效率。
模型调优与验证
基于多维评估结果优化大模型参数，确保生成代码的准确性和符合需求，为模型训练提供科学依据。
Long-term Vision
工业级持续集成/持续部署（CI/CD）
将该框架嵌入企业软件开发流水线，实现自动化、实时的代码质量监控和反馈，推动AI辅助开发的普及。
Abstract
Large language models (LLMs) have demonstrated strong capabilities in code generation, underscoring the critical need for rigorous and comprehensive evaluation. Existing evaluation approaches fall into three categories, including human-centered, metric-based, and LLM-based. Considering that human-centered approaches are labour-intensive and metric-based ones overly rely on reference answers, LLM-based approaches are gaining increasing attention due to their stronger contextual understanding capabilities. However, they generally evaluate the generated code based on static prompts, and tend to fail for complex code scenarios which typically involve multiple requirements and require more contextual information. In addition, these approaches lack fine-grained evaluation for complex code, resulting in limited explainability. To mitigate the limitations, we propose CodeVisionary, the first agent-based evaluation framework for complex code generation. CodeVisionary consists of two stages: (1) Requirement-guided multi-dimensional context distillation stage and (2) Fine-grained scoring and summarization stage. A comprehensive evaluation report is also generated for enhanced explainability. For validation, we construct a new benchmark consisting of 363 samples spanning 37 coding scenarios and 23 programming languages. Extensive experiments demonstrate that CodeVisionary achieves the best performance among three baselines for evaluating complex code generation, outperforming the best baseline with average improvements of 0.217, 0.163, and 0.141 in Pearson, Spearman, and Kendall-Tau coefficients, respectively. The resources of CodeVisionary are available at https://github.com/Eshe0922/CodeVisionary.