RadOT-Eval: Auditable Structured-Evidence Transport for Radiology Report Evaluation
RadOT-Eval employs structured evidence optimal transport for radiology report evaluation, achieving a Spearman correlation of 0.715 with total errors.
Key Findings
Methodology
RadOT-Eval utilizes an entropy-regularized optimal transport (OT) framework to align attribute-structured clinical evidence units extracted from reports. It decomposes reports into units such as findings, anatomy, polarity, and modifiers, then constructs a ground cost matrix based on attribute-specific costs (e.g., Jaccard similarity, categorical mismatches). The OT plan is computed to optimally align these units, separating stable attributes for alignment and sensitive attributes for discrepancy scoring. A monotone nonnegative risk model then combines OT alignment evidence and side-channel discrepancies (e.g., polarity, device status) to produce error risk scores. Parameters are tuned on ReXVal and evaluated on RadEvalX, demonstrating superior correlation metrics compared to traditional similarity measures and LLM-based evaluators.
Key Results
- RadOT-Eval achieves Spearman correlations of 0.715, 0.548, and 0.399 with total, clinically significant, and insignificant errors, respectively, outperforming BLEU, BERTScore, RadGraph-F1, RadCliQ, and GREEN-radllama2-7B by margins of 0.146 to 0.258.
- In synthetic error robustness tests on ReXErr-v1, it attains AUROC of 0.768 and a paired win rate of 0.990 for corrupted reports, indicating high sensitivity to high-severity errors.
- The approach’s interpretability stems from explicit evidence alignment and attribute-specific discrepancy scoring, enabling offline auditability and model development transparency.
Significance
This work advances automatic evaluation by moving beyond surface similarity metrics, addressing the critical need for factual and clinical error detection in high-stakes medical text. Its structured evidence transport framework enhances interpretability, robustness, and auditability, crucial for deploying AI in clinical settings. The method bridges the gap between black-box LLM evaluations and traditional rule-based metrics, offering a scalable, explainable tool for model validation and regulatory compliance, ultimately fostering safer AI integration into healthcare workflows.
Technical Contribution
The core technical innovation lies in modeling report evaluation as evidence transport over attribute-structured units, leveraging entropy-regularized OT for stable alignment. The separation of stable attributes for alignment and sensitive attributes for discrepancy scoring introduces a novel modular approach. Incorporating monotone risk readout ensures risk scores are interpretable and non-decreasing with evidence discrepancies. The framework supports partial and unbalanced OT variants, enhancing flexibility and robustness. These contributions collectively enable a transparent, rank-oriented, and clinically meaningful evaluation pipeline.
Novelty
This is the first application of attribute-based evidence transport in radiology report evaluation, explicitly modeling stable attributes for alignment and using a monotone risk model for error ranking. Unlike prior methods relying solely on lexical similarity or embedding distances, RadOT-Eval emphasizes structured, attribute-specific evidence, providing both interpretability and robustness. Its integration of OT with clinical attribute costs and side-channel discrepancy scoring marks a significant departure from existing similarity-based or LLM-only evaluators, establishing a new paradigm for high-stakes text assessment.
Limitations
- The method depends on predefined attribute schemas and cost parameters, which may require adaptation across different report types or clinical domains, limiting generalizability.
- Handling report ambiguity, multi-finding co-occurrence, and complex language nuances remains challenging, potentially affecting accuracy in atypical cases.
- Computational overhead from OT calculations, especially with large datasets or real-time applications, may hinder scalability without further optimization.
Future Work
Future directions include dynamic learning of attribute weights, integrating multi-modal data such as imaging features, and extending the framework to other clinical documentation types. Enhancing robustness to report ambiguity and rare errors, optimizing computational efficiency, and incorporating clinician feedback for improved interpretability are also key goals. These efforts aim to facilitate broader clinical adoption and support regulatory validation of AI systems in healthcare.
AI Executive Summary
Evaluating the quality of automatically generated radiology reports is critical for safe clinical deployment. Traditional metrics like BLEU or ROUGE focus on superficial text overlap, often missing factual inaccuracies or unsupported findings. This gap poses a significant challenge in high-stakes medical contexts where errors can impact patient care. To address this, RadOT-Eval introduces a novel structured evidence transport framework that decomposes reports into attribute-specific clinical units, such as findings, anatomy, and polarity. Using an entropy-regularized optimal transport algorithm, it aligns these units based on stable attributes, separating them from sensitive discrepancy attributes. This design allows the model to produce rank-oriented error risk scores that are both interpretable and auditable.
The system was trained and tuned on the ReXVal dataset, which provides expert annotations, and evaluated on the independent RadEvalX dataset. Results show that RadOT-Eval achieves a Spearman correlation of 0.715 with total errors, surpassing traditional metrics and LLM-based evaluators like GREEN-radllama2-7B. Its robustness was confirmed through synthetic error tests, where it achieved an AUROC of 0.768 and a paired win rate of 0.990 against corrupted reports. These findings demonstrate that structured evidence transport offers a transparent, reliable, and clinically meaningful evaluation tool, essential for high-stakes AI deployment.
This approach enhances model development workflows by providing detailed, interpretable error assessments, enabling offline audits before clinical deployment. Its modular design supports future extensions, such as multi-modal integration and adaptive attribute weighting, to further improve accuracy and robustness. Despite current limitations related to schema dependence and computational costs, RadOT-Eval sets a new standard for explainable, high-fidelity evaluation of clinical text generation, promising safer and more trustworthy AI systems in healthcare.
Deep Analysis
Background
近年来，医疗AI在放射学中的应用逐步成熟，自动报告生成和评估成为研究热点。早期指标如BLEU、ROUGE主要关注文本表面相似性，难以反映临床事实的正确性。结构化评估方法如RadGraph、RadCliQ引入实体关系和标签，但仍缺乏对局部证据的可追溯性。近年来，基于大模型的评估方法如GREEN逐渐兴起，能提供临床导向的判断，但缺乏稳定的局部对齐机制。高风险场景要求评估工具具备可解释性、鲁棒性和可审计性，推动了结构证据传输和单调风险模型的研究。
Core Problem
现有指标难以同时满足临床事实一致性、错误定位和模型可解释性需求。词级或嵌入级相似性指标无法提供局部对齐信息，模型鲁棒性不足，难以应对复杂报告中的多样性和模糊性。高风险场景下的错误检测仍面临挑战，尤其在临床审计和模型调优中缺乏可追溯的证据链。如何构建一种既具备结构化、可解释性，又能稳健捕获临床错误的评估框架，成为亟待解决的问题。
Innovation
RadOT-Eval的核心创新在于引入结构化临床证据单元作为传输对象，定义稳定属性用于对齐，敏感属性用于差异检测，结合单调风险模型实现错误风险排序。其主要创新点包括：• 基于属性的证据拆解，增强局部可追溯性；• 采用熵正则化的最优传输，支持部分和非平衡传输，提升鲁棒性；• 结合单调风险模型，确保风险评分的可解释性和稳定性。该方法区别于传统的词或嵌入相似性指标，提供了更符合临床逻辑的评估机制，为高风险场景下的模型审计提供了新思路。
Methodology
- �� 单元提取：利用预训练的Llama-3.3-70B-Instruct解析报告，拆解为属性化临床单元（发现、解剖、极性、文本等）；• 证据对齐：定义属性特定的匹配成本（如Jaccard相似、类别匹配等），构建地面成本矩阵；• 最优传输：采用熵正则化的Sinkhorn算法，计算平衡或非平衡的传输计划，确保对齐的稳定性；• 差异检测：在对齐基础上，计算侧信道差异（如极性、设备状态等），作为风险特征；• 单调风险模型：将对齐和差异特征输入非负最小二乘，输出排序的错误风险评分。整个流程在ReXVal调优后，应用于RadEvalX进行验证。
Experiments
采用ReXVal作为调优数据集，调节传输参数和特征权重，最终在RadEvalX上评估性能。对比基线包括BLEU、BERTScore、RadGraph-F1、RadCliQ和绿色模型，使用Spearman相关性作为主要指标。还在ReXErr-v1合成干扰数据上测试鲁棒性，分析不同属性对错误检测的贡献。参数调优通过五折交叉验证实现，确保模型泛化能力。实验还包括不同传输变体（部分、非平衡）以验证方法的稳健性。
Results
RadOT-Eval在RadEvalX上对总错误和临床显著错误的相关性分别达0.715和0.548，显著优于传统指标和绿色模型，提升0.146和0.115。合成干扰测试中，AUROC达0.768，干扰报告胜率0.990，显示其对高风险错误的敏感性。特征分析表明，结构化差异对高严重度错误贡献最大，模型鲁棒性强，能有效捕获复杂报告中的临床事实偏差。
Applications
该方法适用于放射学报告自动评估、模型调优和临床审计，能为模型开发提供可追溯的错误风险排序，提升模型可信度。未来可结合多模态信息，扩展到其他医学文本和影像任务，推动AI在临床决策中的应用。
Limitations & Outlook
模型依赖预定义属性和成本参数，可能在不同临床场景下需要调优，存在迁移限制。对多义性和模糊性处理有限，复杂异常检测仍需改进。计算成本较高，未来需优化效率和扩展能力。
Plain Language Accessible to non-experts
想象你在一家工厂里，工厂每天都要检查生产的商品是否符合标准。传统的方法就像用肉眼简单比对两个商品的外观，容易漏掉一些细节或误判。而RadOT-Eval就像用一台智能检测仪，把商品拆解成不同的部分（比如颜色、大小、标签），然后用数学工具（类似于传输计划）把这些部分一一对应，找到哪里不一样。这个检测仪还能告诉你哪些差异最重要，比如颜色变化或标签错误，帮助你快速定位问题。这样一来，不仅能更准确地检测出商品的瑕疵，还能提供详细的证据链，让工厂管理者可以追溯每个问题的根源。这个方法就像给工厂装上了一个智能的“眼睛”，让检测变得更科学、更透明，也更可靠。
ELI14 Explained like you're 14
想象你在玩一个拼图游戏，你有一张完整的图片（参考报告）和一张你自己拼的（候选报告）。传统的评分就像数你拼对的块数，但不能告诉你哪里拼错了。RadOT-Eval就像用一个聪明的机器人，它会把每个拼图块拆开，找出每个块对应的正确位置，然后用数学方法（叫做最优传输）把它们一一匹配。这个机器人还会特别关注一些重要的细节，比如颜色是否一样、标签是否正确、拼图块的边缘是否对齐。最后，它会给出一个“风险分数”，告诉你这张拼图的整体错误有多大，哪些地方最可能出错。这样一来，你就可以清楚看到拼图哪里出了问题，为什么出问题，甚至可以追溯到拼图的每一块。这就像给拼图游戏装上了智能检测器，让你更快、更准地找到拼错的地方，确保拼图拼得完美无瑕。
Abstract
Automatic evaluation is critical for high-stakes text generation, where errors often involve omitted findings, hallucinated content, polarity reversals, location changes, uncertainty mismatches, and temporal-comparison errors rather than low surface similarity alone. Radiology report generation provides a challenging test case because generated reports must preserve structured clinical evidence across sources. We present RadOT-Eval, an interpretable structured-evidence optimal transport framework for offline auditing of radiology report generation. RadOT-Eval decomposes reference and candidate reports into attribute-structured clinical evidence units, aligns corresponding evidence using entropy-regularized optimal transport, and uses clinically meaningful side-channel discrepancies in a monotone risk model to predict error burden. All transport, feature, and readout choices are selected using the ReXVal dataset, and the frozen system is evaluated on the independent RadEvalX dataset. RadOT-Eval achieves Spearman correlations of 0.715, 0.548, and 0.399 with total, clinically significant, and clinically insignificant annotated error burden, respectively, yielding higher point estimates than standard evaluation metrics and the open-source large language model (LLM)-based evaluator GREEN-radllama2-7B. In a frozen auxiliary corruption-sensitivity stress test on ReXErr-v1, RadOT-Eval achieves 0.768 AUROC and a 0.990 corrupted-greater-than-clean paired win rate. These results show that structured evidence transport provides an auditable, rank-oriented evaluation tool for high-stakes generated clinical text under ReXVal-only model selection and frozen RadEvalX testing.