SynthEHR-Eviction: Enhancing Eviction SDoH Detection with LLM-Augmented Synthetic EHR Data

The evolution of personalized medicine emphasizes incorporating social determinants of health (SDoH) into clinical decision-making. Prior works like MIMIC-IV and NLP rule-based systems have highlighted the importance but face limitations: sparse documentation, lack of standardized coding (e.g., ICD-10), and high annotation costs. Recent advances in large language models (LLMs) such as GPT-4, LLaMA, and Qwen have shown potential in automating information extraction, yet their application to complex, underrepresented SDoH domains remains limited. Addressing social risks like eviction requires high-quality, scalable annotation methods to facilitate integration into routine healthcare workflows. This background underscores the need for innovative solutions that combine synthetic data, prompt engineering, and expert validation to overcome existing bottlenecks.

Detecting eviction status within clinical notes is hindered by sparse, unstructured data and the absence of specific codes. Traditional rule-based or small-scale models are costly and lack scalability. Existing datasets are limited, preventing robust model training. The core challenge is developing a scalable, accurate, and generalizable method to identify nuanced eviction states, such as 'pending' or 'historical,' across diverse clinical settings, thereby enabling timely social interventions and reducing health disparities.

The study introduces several innovations: 1) leveraging large language models for synthetic data generation, 2) implementing automated prompt optimization (APO) to refine model prompts iteratively, 3) integrating human-in-the-loop validation via DSPy for chain-of-thought reasoning, 4) constructing the largest annotated eviction-related SDoH dataset with 14 categories, and 5) enabling multi-scale model fine-tuning. These advances address data scarcity, improve detection accuracy, and reduce annotation costs, setting a new benchmark for social risk extraction in clinical NLP.

• �� Data augmentation: Use large models (Qwen2.5, LLaMA3) with prompts refined through APO to generate synthetic clinical notes covering 14 eviction categories.
• �� Prompt optimization: Iteratively adjust prompts based on model performance and expert feedback, enhancing alignment with clinical language.
• �� Validation: Experts verify generated samples, employing DSPy’s chain-of-thought reasoning to ensure label accuracy.
• �� Dataset construction: Combine synthetic and real notes from MIMIC-IV and PMC, balancing class distributions.
• �� Model fine-tuning: Use augmented data to train models like Qwen2.5-7B, BioBERT, and LLaMA3.
• �� Evaluation: Measure performance via Macro-F1, Micro-F1, across binary, multi-class eviction, and non-eviction categories, with cross-dataset validation.

The experimental setup involved training models on a dataset of 8,000 synthetic and 616 expert-annotated notes, testing on separate real-world notes from MIMIC-IV and PMC. Baselines included GPT-4o, BioBERT, and LLaMA variants. Metrics such as F1, precision, recall evaluated binary eviction detection, multi-class fine-grained classification, and broader SDoH categories. Ablation studies assessed APO’s impact, and cross-dataset tests evaluated generalization. Hyperparameters were tuned for prompt refinement and model training, ensuring robustness across scenarios.

Fine-tuned Qwen2.5-7B achieved 88.8% Macro-F1 on eviction detection, outperforming GPT-4o-APO (87.8%). Multi-class classification saw APO improve GPT-4o-mini’s Macro-F1 from 69.1% to 78.1%. Models like LLaMA-3.1-8B and Qwen2.5-7B maintained scores above 88%, demonstrating scalability. The synthetic dataset enabled effective transfer learning, with models generalizing well across datasets. Category-specific F1 scores reached up to 96%, validating nuanced detection capabilities.

The approach can be integrated into clinical decision support systems to automatically flag eviction risks, aiding social workers and policymakers. It supports proactive interventions, resource allocation, and targeted outreach, especially for vulnerable populations. The scalable pipeline facilitates continuous monitoring of social risks, improving health equity. Long-term, it can underpin comprehensive SDoH analytics platforms, informing public health policies and resource planning.

Despite high performance, models struggle with ambiguous language and rare categories. Synthetic data may introduce biases affecting real-world accuracy. Computational demands of large models limit deployment in resource-constrained settings. Fully automated, bias-free solutions remain a future goal. Further research is needed to validate models across diverse populations and clinical environments.