EHR-Complex: Benchmarking Medical Agents for Complex Clinical Reasoning
EHR-Complex is a large-scale interactive clinical reasoning benchmark based on MIMIC-IV, with an average SQL complexity of 31.93 components and top model accuracy of 62.3%.
Key Findings
Methodology
This study constructs EHR-Complex leveraging MIMIC-IV, employing patient event graphs, clinical evidence paths, and SQL templates to enable multi-turn interaction. Tasks span patient and cohort levels, emphasizing longitudinal multi-table aggregation and compositional reasoning. Models execute SQL or Python, receiving environment feedback, to verify answers. Twelve state-of-the-art models are evaluated, revealing a maximum accuracy of 62.3% and Pass^k consistency below 50% at k=4, exposing stochastic fragility. The framework integrates real-world SQL complexity, with an average of 31.93 components per query, to simulate real clinical reasoning challenges.
Key Results
- The best model achieves only 62.3% exact match, indicating high complexity compared to static benchmarks near 100%.
- Performance drops from 0.85 to 0.25 at the cohort level, highlighting challenges in multi-patient alignment and aggregation.
- Failure modes include SQL logic errors (37.9%), medical code lookup failures (28.6%), and semantic misunderstandings (23.1%), emphasizing reasoning and knowledge gaps.
Significance
This benchmark pioneers large-scale, interactive evaluation of clinical reasoning, bridging the gap between static datasets and real-world complexity. It advances AI's capacity for dynamic, multi-step, multi-table inference in healthcare, addressing critical bottlenecks in deploying AI for clinical decision support. By exposing current models' limitations, it guides future research toward more robust, explainable, and clinically reliable AI systems, fostering progress in automated EHR analysis and intelligent healthcare delivery.
Technical Contribution
The framework combines patient event graphs, evidence path sampling, and SQL template synthesis, enabling multi-round interaction with environment feedback. It introduces a novel complexity metric based on SQL structural components, significantly increasing task difficulty. The approach supports multi-level reasoning, robustness analysis, and failure mode identification, setting a new standard for evaluating AI in complex clinical environments. This integration of structured data, multi-step reasoning, and environment feedback is a key technical advancement over prior static or simplistic benchmarks.
Novelty
This is the first large-scale, interactive benchmark based on real EHR data emphasizing longitudinal multi-table reasoning and compositionality. Unlike previous static datasets, it supports multi-turn environment interaction, simulating real clinical workflows. The use of patient event graphs and evidence paths for dynamic SQL generation represents a significant innovation, providing a more realistic and challenging testbed for AI models in healthcare.
Limitations
- Models still struggle with complex logic, knowledge integration, and semantic understanding, often making errors in reasoning steps.
- The benchmark environment is simulated, lacking full clinical noise and variability, which may limit real-world applicability.
- Performance on cohort-level tasks remains fragile, especially in multi-patient alignment and aggregation, indicating robustness issues that need addressing.
Future Work
Future research will incorporate knowledge graphs and reinforcement learning to enhance reasoning and knowledge fusion. Expanding modalities like imaging and text will improve clinical realism. Improving model robustness, interpretability, and efficiency remains a priority. Additionally, integrating real-world noisy data and deploying models in clinical settings will be explored to bridge the gap between research and practice.
AI Executive Summary
Electronic health records (EHR) are vital for modern healthcare, yet their complexity poses significant challenges for AI systems. Existing benchmarks largely focus on static question-answering or simple SQL generation, which do not capture the dynamic, multi-step reasoning required in real clinical environments. To address this gap, we introduce EHR-Complex, a comprehensive benchmark built on the MIMIC-IV dataset, designed to evaluate AI agents in interactive, multi-turn clinical reasoning tasks.
EHR-Complex employs a novel framework integrating patient event graphs, clinical evidence paths, and SQL templates, enabling models to perform multi-step reasoning with environment feedback. Tasks are crafted to reflect real-world complexity, with an average of 31.93 SQL structural components per query, emphasizing longitudinal multi-table aggregation and compositional reasoning. The benchmark covers six clinical intents at both patient and cohort levels, supporting diverse information needs.
Evaluation of 12 state-of-the-art large language models reveals that even the best achieves only 62.3% accuracy, with Pass^k consistency below 50% at k=4. This highlights the significant gap between current AI capabilities and the demands of reliable clinical reasoning. Failure analysis points to logic errors, medical code lookup failures, and semantic misunderstandings as primary challenges. These insights underscore the need for more robust, explainable AI models tailored for complex healthcare tasks.
Overall, EHR-Complex sets a new standard for evaluating AI in healthcare, pushing models toward greater robustness and interpretability. Its design encourages future innovations in multi-modal, multi-step reasoning, ultimately aiming to facilitate safer, more effective AI-assisted clinical decision-making. Despite its advances, the benchmark also reveals persistent limitations, such as sensitivity to environment variability and multi-patient alignment, guiding future research directions.
Deep Dive
Plain Language Accessible to non-experts
想象你在一家大型厨房里，厨师需要准备一道复杂的菜肴。每个步骤都依赖前面做好的材料和调料，还要根据现场的情况灵活应对，比如调味不足或食材变质。这个厨房里有很多不同的厨房用具和食材，每次操作都需要检查和调整。厨师不仅要记住每个步骤，还要根据现场反馈调整操作，比如重新查询或修改菜谱。现在，把这个厨房比作医院的电子健康记录系统，厨师就是AI模型，菜肴就是需要解决的临床问题。模型要像厨师一样，理解不同的材料（患者信息、医学编码），根据现场反馈（环境信息）调整操作（SQL或Python代码），最终做出正确的菜肴（答案）。这个过程非常复杂，需要多次尝试和调整，才能做出令人满意的结果。这个基准就像是模拟厨房的训练场，让模型学会在复杂环境中灵活应对各种挑战。
ELI14 Explained like you're 14
想象你在学校的厨房帮忙做饭。这里有很多不同的食材、调料和工具，每次做菜都要按照步骤来，但有时候你会发现食材不够或者味道不对，需要不断试错和调整。比如，你要做一道汤，先放菜，然后试味道，觉得不够咸就再放盐。这个过程很像模型在处理医院里的电子病历，它要理解很多不同的资料，比如病人的信息、检查报告、药物记录，还要根据环境反馈调整操作，比如重新查询或修改SQL语句。这个基准就像是模拟厨房的训练场，让模型学会在复杂的环境中不断试错，最终做出正确的菜。这很难，但也很有趣，因为它让AI变得更聪明、更像人类厨师，能应对各种突发情况。未来，这样的模型可以帮医生更快找到病人的信息，做出更好的诊断，就像一个超级厨师一样厉害！
Abstract
Clinical agents promise to democratize access to electronic health records (EHRs), yet existing benchmarks fail to reflect the complexity of practical EHR analysis, e.g., often operating on idealized, clean EHRs via static SQL generation rather than interactive execution. In this work, we introduce EHR-Complex, a large-scale benchmark designed for interactive clinical database reasoning. Built on the large MIMIC-IV substrate (365K patients, 31 tables, 500M+ records), EHR-Complex comprises about 52K tasks spanning six clinical intents, supporting both patient-level and population-level queries, where each task requires an agent to interact with a sandboxed environment by executing SQL queries or Python code. Notably, EHR-Complex considers the real-world SQL task complexity for longitudinal multi-table aggregation and compositional reasoning, resulting in 31.93 SQL structural components per query on average. Evaluation results on EHR-Complex reveal the clinical difficulty of these EHR reasoning scenarios, with the top-performing model achieving only 62.3% exact-match accuracy. Pass^k consistency drops below 50% for nearly all evaluated models at k=4, exposing broad stochastic fragility. A fine-grained analysis of more than 3,800 failed trajectories for representative LLMs reveals three dominant failure modes: SQL logic errors, medical-code lookup failures, and semantic misunderstandings. EHR-Complex provides a rigorous testbed for clinical agents and highlights remaining gaps in robust reasoning for large-scale EHR analysis.