JudgeFlow: Agentic Workflow Optimization via Block Judge
JudgeFlow uses block responsibility scores to optimize LLM workflows, improving efficiency and interpretability.
Key Findings
Methodology
This paper introduces JudgeFlow, integrating reusable logic blocks (sequential, loop, conditional) with a dedicated Judge module that analyzes execution traces, especially failures, to assign responsibility scores to blocks. The core algorithm involves ranking blocks based on failure responsibility, guiding targeted modifications via an LLM optimizer. The process iterates through evaluation, diagnosis, and refinement, significantly improving sample efficiency and interpretability. The framework supports complex control flows and is validated on mathematical reasoning (GSM8K, MATH) and code generation (MBPP, HumanEval), outperforming existing methods with an average performance increase of 1.4%.
Key Results
- On benchmarks like GSM8K, MATH, MBPP, and HumanEval, JudgeFlow surpasses prior approaches, with an average score increase of 1.4%, notably improving performance on challenging datasets such as MATH (+3.1%). In AIME 2025, the average score reached 44.67, demonstrating robustness.
- Block-level fault diagnosis accelerates convergence, with rapid early improvements and stable final performance. The responsibility ranking effectively guides the optimizer, reducing search space and improving targeted modifications.
- Model capacity impacts results: using GPT-4, Gemini-2.5, larger models yield better scores. Cross-task transfer tests show strong generalization, validating the framework’s versatility across reasoning and coding tasks.
Significance
This work addresses the limitations of coarse, end-to-end evaluation signals by introducing a fine-grained, block-level responsibility mechanism. It enhances the efficiency, interpretability, and scalability of AI workflow optimization, crucial for deploying reliable, autonomous AI systems in complex real-world scenarios. The approach paves the way for more intelligent, self-improving agents capable of handling intricate tasks with minimal human intervention, impacting both academia and industry.
Technical Contribution
The key technical innovation lies in the integration of a responsibility scoring algorithm with logic block abstraction, enabling precise fault localization. The framework combines multi-round iterative refinement with block-level modifications, supported by a novel diagnosis-guided optimizer. This approach extends traditional end-to-end evaluation by providing intermediate, interpretable signals, facilitating scalable and effective workflow optimization in complex environments.
Novelty
This is the first framework to incorporate responsibility-based fault attribution at the block level within a structured workflow, enabling targeted, fine-grained optimization. Unlike prior methods relying solely on global evaluation, JudgeFlow’s responsibility ranking offers a new paradigm for scalable, interpretable AI system tuning, especially in complex, multi-control flow scenarios.
Limitations
- Reliance on LLMs for responsibility scoring introduces bias and potential misdiagnosis, especially in ambiguous failure cases, which can mislead the optimizer.
- Computational costs are high due to multiple iterations and large models, limiting real-time or large-scale deployment.
- Handling highly dynamic or unpredictable control structures remains challenging, as responsibility attribution may become less accurate in such scenarios.
Future Work
Future research will focus on improving fault diagnosis robustness, possibly integrating multi-modal signals. Efforts will also aim to reduce computational overhead, explore reinforcement learning for adaptive responsibility scoring, and extend framework applicability to more complex, real-world workflows with dynamic control flows.
AI Executive Summary
JudgeFlow introduces a novel approach to optimizing large language model (LLM) workflows by leveraging block-level responsibility scores. Traditional methods rely on coarse, end-to-end evaluation signals, which often obscure the specific points of failure, leading to inefficient and less interpretable optimization. In contrast, JudgeFlow constructs workflows from structured logic blocks—sequential, loop, and conditional—and employs a dedicated Judge module that analyzes execution traces, especially failures, to assign responsibility scores to individual blocks. This fine-grained diagnostic process enables targeted modifications, such as adding, removing, or reconfiguring blocks, guided by an LLM optimizer.
The framework iterates through evaluation, diagnosis, and refinement, significantly improving sample efficiency and interpretability. Experimental results on benchmarks like GSM8K, MATH, MBPP, and HumanEval demonstrate that JudgeFlow outperforms existing methods, with an average performance increase of 1.4%, and achieves superior results on challenging datasets such as MATH (+3.1%). The responsibility scoring mechanism effectively guides the optimizer, leading to rapid early improvements and stable convergence.
This work represents a significant step forward in automating complex AI workflows, addressing the limitations of global evaluation signals. By enabling precise fault localization and targeted optimization, JudgeFlow enhances the reliability and scalability of autonomous AI systems. The approach’s adaptability across reasoning and coding tasks, along with its potential for extension to more complex control structures, underscores its broad impact. Despite current limitations like computational costs and bias risks, ongoing research aims to refine responsibility attribution and reduce resource demands, paving the way for more intelligent, self-improving agents in the future.
Deep Analysis
Background
随着大规模预训练模型（LLMs）的快速发展，智能代理系统在自然语言理解、推理、代码生成等领域取得了显著突破。早期工作如Chain-of-Thought（Wei et al., 2023）和多智能体系统（如MetaGPT、AutoGen）推动了复杂任务的解决能力，但仍依赖手工设计的流程和调优策略，存在成本高、灵活性差的问题。AutoML在传统机器学习中的成功激发了在LLM工作流中的自动调优探索，然而大部分方法仍局限于端到端性能优化，缺乏对中间环节的细粒度故障诊断，难以实现高效、可解释的调优。近年来，研究逐渐关注结构化工作流的自动优化，尝试引入图结构、代码表示等多种抽象方式，但面临复杂控制流中的故障定位难题。
Core Problem
核心问题在于复杂工作流中的故障定位与责任归因。传统方法依赖整体评价指标，难以识别具体哪个环节出错，导致调优效率低下。复杂控制结构（如条件分支、循环）中的错误难以追踪，影响调优效果。此外，探索空间大，调优过程中的搜索成本高，缺乏有效的指导机制。解决这一问题对于提升AI系统的自主性和可靠性具有重要意义，尤其在多任务、多模型环境中，如何实现细粒度、可解释的故障定位成为关键。
Innovation
本研究提出结合责任评分的块级故障诊断机制，创新点包括：1）引入逻辑块（顺序、循环、条件）作为结构化表示，支持复杂控制流；2）设计专门的Judge模块，分析失败轨迹，赋予每个块责任等级，实现细粒度故障定位；3）采用责任排序算法（基于责任分数）引导有针对性的调优策略，结合多轮迭代逐步优化工作流。这些创新共同提升了调优效率，支持复杂控制结构的自动调节，显著优于传统端到端方法。
Methodology
- �� 构建逻辑块（顺序、循环、条件）作为工作流的基本单元，定义其输入、输出和控制参数。
- �� 利用LLM生成操作，形成完整工作流结构。
- �� 设计Judge模块，分析失败案例，采用责任排序算法（如责任分数排序）定位关键故障块。
- �� 根据责任排序结果，选择最有问题的块，利用LLM进行有针对性的调优（添加、删除、修改块）。
- �� 多轮迭代：每次优化后重新评估，逐步提升整体性能。
- �� 实验采用GSM8K、MATH、MBPP、HumanEval等数据集，比较不同调优策略和模型容量的影响。
Experiments
采用公开数学推理和代码生成基准，验证JudgeFlow的性能。设置20轮优化，调优参数（最大块数3、阈值1、保留Top-3工作流）。对比基线包括传统端到端和无责任评分调优方法。指标包括准确率、pass@1等，验证不同模型（GPT-4、Gemini-2.5）对性能的影响。还进行了消融实验，验证块级责任评分的贡献。跨任务迁移测试显示模型具有良好的泛化能力，验证了方法的普适性。
Results
JudgeFlow在数学推理任务中平均提升1.4%，在难度较高的MATH任务中提升3.1%。在代码生成任务中，pass@1指标提升1.8%。多轮优化后，性能快速提升，最终优于无责任评分的对比方法。责任排序机制显著缩短调优时间，提高调优的针对性。跨任务迁移实验表明，模型在不同任务间具有较强的泛化能力，验证了方法的适应性。
Applications
该方法适用于自动调优复杂AI工作流，特别在数学推理、代码生成、决策系统中。可用于科研中的模型调试，也适合工业界的智能系统维护。通过责任评分实现故障定位，降低人工干预成本，提升系统可靠性。未来可结合多模态信息，拓展到更复杂的控制结构和动态环境。
Limitations & Outlook
责任排序可能受模型偏差影响，存在误判风险，尤其在极端或复杂场景中表现不佳。工作流结构复杂或动态变化时，责任归因可能不够准确，影响调优效果。算法计算成本较高，特别在大规模、多轮优化中存在性能瓶颈。未来需提升责任归因鲁棒性，降低成本，增强适应性。
Plain Language Accessible to non-experts
想象你在厨房做菜，整个流程像一份菜谱，每个步骤（逻辑块）包括切菜、炒菜、调味等。偶尔会出现问题，比如炒糊或调味不够。JudgeFlow就像一个聪明的厨师助手，它会观察每个步骤，判断哪个环节出了问题，然后告诉你该改进哪一部分。通过不断调整，菜的味道越来越好。这种方法让厨师（AI）能更快找到问题，做出更美味的菜。它用责任评分的方式，像给每个步骤打分，帮你知道哪里需要改进，整个厨房效率和菜品质量都提升了。
ELI14 Explained like you're 14
想象你在学校做科学实验，整个流程像一条长长的链条。有时候会出错，比如实验失败或步骤不对。JudgeFlow就像一个聪明的老师，能帮你找出哪个步骤出了问题。它会观察你的实验过程，给每个步骤打分，告诉你哪个地方最需要改进。比如，可能是你没有正确混合材料，老师会建议你改正。这样，你就可以专注于修正最关键的部分，整个实验就会更成功。这个方法让你不用反复试错，而是有针对性地改进每个环节，效率大大提高，也更容易理解哪里出了问题。
Glossary
责任评分 (Responsibility Scoring)
一种根据模型输出分析故障责任的算法，帮助定位问题块。In this paper, it assigns responsibility scores to code blocks based on failure analysis.
用于Judge模块中，识别出最可能导致失败的逻辑块。
逻辑块 (Logic Block)
构建工作流的基本结构单元，包括顺序、循环和条件块。It abstracts control structures in code-based workflows.
作为工作流的中间表示，支持可解释性和调优。
责任排序 (Responsibility Ranking)
基于责任分数对块进行排序，优先修正最有问题的块。It通过责任排序引导优化流程。
在调优阶段用于选择重点改进对象。
工作流 (Workflow)
由多个逻辑块组成的执行序列，用于完成特定任务。It可以是代码、图结构或神经网络。
模型的整体执行结构。
责任归因 (Fault Attribution)
分析失败原因，将责任归结到具体的工作流块或步骤。It是调优的基础。
Judge模块的核心功能。
Abstract
Optimizing LLM-based agentic workflows is challenging for scaling AI capabilities. Current methods rely on coarse, end-to-end evaluation signals and lack fine-grained signals on where to refine, often resulting in inefficient or low-impact modifications. To address these limitations, we propose JudgeFlow, an Evaluation-Judge-Optimization-Update pipeline. We incorporate reusable, configurable logic blocks into agentic workflows to capture fundamental forms of logic. On top of this abstraction, we design a dedicated Judge module that inspects execution traces particularly failed runs and assigns rank-based responsibility scores to problematic blocks. These fine-grained diagnostic signals are then leveraged by an LLM-based optimizer, which focuses modifications on the most problematic block in the workflow. Our approach improves sample efficiency, enhances interpretability through block-level diagnostics, and provides a scalable foundation for automating increasingly complex agentic workflows. We evaluate JudgeFlow on mathematical reasoning and code generation benchmarks, where JudgeFlow achieves superior performance and efficiency compared to existing methods.