FHIR-AgentBench: Benchmarking LLM Agents for Realistic Interoperable EHR Question Answering
FHIR-AgentBench evaluates LLM performance in EHR QA under HL7 FHIR, highlighting data retrieval and reasoning challenges.
Key Findings
Methodology
This study introduces the FHIR-AgentBench benchmark to evaluate LLMs in clinical QA under HL7 FHIR standards. By comparing different data retrieval strategies (direct FHIR API calls vs. specialized tools), interaction patterns (single-turn vs. multi-turn), and reasoning strategies (natural language vs. code generation), it reveals practical challenges in data retrieval and reasoning.
Key Results
- Multi-turn interaction significantly increased retrieval recall from 58% to 71%.
- Code generation significantly improved answer correctness, with multi-turn code generation achieving 50% correctness.
- Retrieval precision was consistently low, introducing noise that affected reasoning capabilities.
Significance
This study provides a realistic evaluation framework for clinical AI, promoting the application of LLMs in complex FHIR resources. By publicly releasing the dataset and evaluation suite, it fosters reproducible research and the development of reliable clinical applications.
Technical Contribution
This study is the first to combine real clinical questions with FHIR resources, providing a comprehensive evaluation framework. By systematically comparing different agent frameworks, it reveals the critical relationship between retrieval precision and answer correctness.
Novelty
FHIR-AgentBench is the first to combine real clinical questions with FHIR resources, providing a comprehensive evaluation framework that fills the gap in interoperability assessment.
Limitations
- Low retrieval precision introduces noise, affecting reasoning capabilities.
- Single-turn agents cannot handle multi-resource type questions.
Future Work
Future research can explore methods to improve retrieval precision and develop more complex agent architectures to enhance reasoning capabilities.
AI Executive Summary
FHIR-AgentBench is a new benchmark for evaluating large language models (LLMs) in electronic health record (EHR) question answering under the HL7 FHIR standard. Existing benchmarks fail to fully reflect the complexity of the FHIR standard, leading to a lack of realism in LLM evaluation.
The study compares different data retrieval strategies, interaction patterns, and reasoning strategies, revealing practical challenges in data retrieval and reasoning. Experimental results show that multi-turn interaction significantly improves retrieval recall, while code generation significantly enhances answer correctness.
The public release of FHIR-AgentBench promotes reproducible research and reliable clinical application development, providing a realistic evaluation framework for clinical AI. Future research can explore methods to improve retrieval precision and develop more complex agent architectures to enhance reasoning capabilities.
Deep Analysis
Background
With the widespread adoption of the HL7 FHIR standard, clinical AI faces new challenges. Traditional structured data retrieval methods are no longer applicable, necessitating new evaluation frameworks to reflect the complexity of the FHIR standard.
Core Problem
Existing EHR QA benchmarks fail to fully reflect the complexity of the FHIR standard, leading to a lack of realism in LLM evaluation. A new benchmark is needed to assess LLMs' QA capabilities on complex FHIR resources.
Innovation
FHIR-AgentBench combines real clinical questions with FHIR resources, providing a comprehensive evaluation framework. By systematically comparing different agent frameworks, it reveals the critical relationship between retrieval precision and answer correctness.
Methodology
- �� Compare different data retrieval strategies: direct FHIR API calls vs. specialized tools.
- �� Compare different interaction patterns: single-turn vs. multi-turn.
- �� Compare different reasoning strategies: natural language vs. code generation.
Experiments
Using the MIMIC-IV-FHIR dataset, evaluate different agent frameworks on retrieval precision, recall, and answer correctness. Improve answer correctness through multi-turn interaction and code generation.
Results
Multi-turn interaction significantly increased retrieval recall, while code generation significantly enhanced answer correctness. Retrieval precision was consistently low, introducing noise that affected reasoning capabilities.
Applications
The benchmark can be used to evaluate LLM performance in clinical QA, promoting the development and application of clinical AI.
Limitations & Outlook
Low retrieval precision introduces noise, affecting reasoning capabilities. Single-turn agents cannot handle multi-resource type questions. Future research can explore methods to improve retrieval precision.
Plain Language Accessible to non-experts
Imagine a library with various books on the shelves. You need to find a specific book to answer a question. Traditional methods are like searching the shelves, while FHIR-AgentBench is like a smart assistant helping you quickly find the right book and provide the answer. This assistant can not only find the book but also understand its content and give accurate answers.
ELI14 Explained like you're 14
Imagine playing a complex game where you need to find hidden treasures. This game has many levels, each with different clues. FHIR-AgentBench is like a super assistant helping you find clues, solve puzzles, and eventually find the treasure. It can not only find clues but also help you understand them and give the correct answers.
Glossary
FHIR (Fast Healthcare Interoperability Resources)
A standardized format for exchanging electronic health data, aimed at improving data interoperability.
Used for organizing and querying electronic health data.
LLM (Large Language Model)
A deep learning-based natural language processing model capable of understanding and generating human language.
Used for handling complex clinical QA tasks.
Retrieval Precision
Measures the accuracy of retrieval results, i.e., the proportion of relevant resources among all retrieved resources.
Used to evaluate agent retrieval performance.
Code Generation
Generating code to perform complex reasoning and computation tasks.
Used to parse complex FHIR data structures.
Recall
Measures the completeness of retrieval results, i.e., the proportion of relevant resources among all relevant resources.
Used to evaluate agent retrieval performance.
Open Questions Unanswered questions from this research
- 1 How to improve retrieval precision to reduce noise impact on reasoning?
- 2 How to design more complex agent architectures to enhance reasoning capabilities?
Applications
Immediate Applications
Clinical QA System
Helps doctors quickly access patient information, improving diagnostic efficiency. Requires high precision retrieval capabilities.
Long-term Vision
Intelligent Medical Assistant
Provides comprehensive patient health analysis by integrating multiple data sources. Requires complex reasoning capabilities.
Abstract
The recent shift toward the Health Level Seven Fast Healthcare Interoperability Resources (HL7 FHIR) standard opens a new frontier for clinical AI, demanding LLM agents to navigate complex, resource-based data models instead of conventional structured health data. However, existing benchmarks have lagged behind this transition, lacking the realism needed to evaluate recent LLMs on interoperable clinical data. To bridge this gap, we introduce FHIR-AgentBench, a benchmark that grounds 2,931 real-world clinical questions in the HL7 FHIR standard. Using this benchmark, we systematically evaluate agentic frameworks, comparing different data retrieval strategies (direct FHIR API calls vs. specialized tools), interaction patterns (single-turn vs. multi-turn), and reasoning strategies (natural language vs. code generation). Our experiments highlight the practical challenges of retrieving data from intricate FHIR resources and the difficulty of reasoning over them, both of which critically affect question answering performance. We publicly release the FHIR-AgentBench dataset and evaluation suite (https://github.com/glee4810/FHIR-AgentBench) to promote reproducible research and the development of robust, reliable LLM agents for clinical applications.