Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settings

The evolution of NLP has led to the widespread adoption of pre-trained language models (PLMs) like BERT, GPT, and their derivatives, which have revolutionized tasks such as summarization, question answering, and translation. In the medical domain, these models promise to automate literature review, clinical documentation, and decision support. However, the specialized vocabulary, frequent neologisms, and abbreviations in medical texts pose significant challenges. Existing models trained on general corpora often encounter high OOV rates when applied directly to medical data, leading to fragmented tokenization and reduced semantic fidelity. Prior efforts, including domain-specific fine-tuning and vocabulary expansion, have shown promise but lack comprehensive evaluation under high OOV and novelty scenarios. This gap limits the deployment of reliable AI tools in clinical settings, where accuracy and trustworthiness are paramount. The current work aims to systematically analyze and improve vocabulary adaptation strategies, bridging this critical gap.

The core issue addressed is the high OOV rate in medical texts, which causes token fragmentation and hampers the model’s understanding and generation capabilities. Existing models, even large ones like Llama-3.1 with 128K tokens, struggle with over-fragmentation of medical terms, leading to degraded summarization quality. The problem is exacerbated in high novelty scenarios where new terms appear in summaries but not in source documents, further challenging the models’ ability to generate faithful outputs. Overcoming these limitations requires effective vocabulary expansion, better tokenization strategies, and domain-specific pretraining, all while maintaining computational efficiency.

The paper introduces a multi-faceted vocabulary adaptation framework that combines candidate vocabulary generation, importance filtering, and embedding training via LoRA. Notably, it proposes ScafFix, which directly selects high-frequency medical terms without relying solely on tokenization-based methods, thereby reducing fragmentation. The integration of these strategies with parameter-efficient continual pretraining enables models to better understand and generate medical summaries, especially in high OOV and novelty scenarios. This approach differs from prior work by directly addressing token fragmentation and leveraging domain-specific corpora for targeted vocabulary expansion, resulting in significant performance gains.

• �� Generate candidate vocabularies from PubMed abstracts and target datasets, focusing on high OOV medical terms.
• �� Filter candidates using fragment scores and frequency to select important tokens.
• �� Expand the model vocabulary by adding selected tokens, employing methods like MEDVOC and ScafFix.
• �� Initialize embeddings of new tokens as averages of existing subword embeddings.
• �� Perform continual pretraining using LoRA, either end-to-end or in two stages, on domain-specific texts.
• �� Fine-tune models on medical summarization datasets, evaluating with Rouge-L and Concept-Score.
• �� Conduct human evaluations to assess relevance and faithfulness of summaries.

The experimental setup involves three datasets—BioASQ, EBM, and PubMedQA—covering query-focused medical summarization tasks. Baselines include models without vocabulary adaptation, with continual pretraining, and with proposed adaptation strategies. Hyperparameters include a batch size of 32, learning rate of 1e-4, with pretraining on 20K PubMed abstracts using LoRA. Evaluation metrics encompass Rouge-L for informativeness and UMLS Concept-Score for faithfulness. Multiple ablation studies compare different vocabulary filtering and training strategies, analyzing performance across high OOV and high novelty thresholds. Results demonstrate consistent improvements with vocabulary adaptation, especially ScafFix, across datasets and scenarios.

The results show that vocabulary adaptation strategies improve model performance in 7/8 high OOV and high novelty scenarios, with average Rouge-L gains of 14-19%. ScafFix consistently outperforms other methods in high OOV contexts, with improvements up to 14.64%. Continual pretraining alone offers benefits but is less effective than combined strategies. Human evaluations confirm that summaries are more relevant and faithful when vocabulary adaptation is employed, indicating better alignment with medical knowledge. These findings validate the importance of targeted vocabulary expansion for domain-specific NLP tasks.

The methods can be directly applied to clinical documentation summarization, medical literature review, and automated knowledge extraction. They enable models to better understand domain-specific terminology, improving accuracy and trustworthiness. Long-term, these strategies could facilitate real-time clinical decision support, personalized medicine, and AI-assisted diagnostics, especially as models adapt to rapidly evolving medical vocabularies and multilingual contexts.

Despite significant improvements, the approach depends on high-quality, domain-specific corpora for vocabulary generation, which may not be available in low-resource languages or emerging fields. The methods still face challenges with extremely rare or newly emerging terms. Computational costs, although reduced via LoRA, remain non-trivial for large-scale deployment. Cross-lingual and multi-modal validation is needed to generalize findings beyond English medical texts. Future work should focus on adaptive, real-time vocabulary refinement and broader domain adaptation.