Seeing Through Multiple Views: Parameter-Efficient Fine-Tuning via Selective Neurons for Consistent Radiology Report Generation
Proposes View-PNDF, a neuron-level fine-tuning method that enhances multi-view X-ray report consistency, achieving state-of-the-art results.
Key Findings
Methodology
This paper introduces the View-PNDF framework, which combines neuron-level analysis to identify and fine-tune view-specific neurons. It includes: • View-specific Neuron Detection (VND), leveraging activation and attribution scores to select neurons responsive to particular views; • Neuron Verification (VNV), perturbation-based assessment quantifying neurons’ impact on report quality; • Selective Fine-tuning (VNF), optimizing only the identified neurons to improve view consistency. The approach analyzes linear and self-attention layers within Transformer decoders, validated on datasets like IU-XRay and MIMIC-CXR.
Key Results
- On IU-XRay and MIMIC-CXR benchmarks, View-PNDF outperforms traditional fusion methods, achieving BLEU-1 scores up to 0.510 (Hulu backbone), ROUGE-L of 0.396, and clinical F1 of 0.414, demonstrating superior multi-view consistency and diagnostic accuracy.
- Fine-tuning involves less than 5% of neurons, reducing computational costs while maintaining generalization. Neuron perturbation experiments show that deactivating view-specific neurons causes significant drops in report quality (e.g., MIMIC-CXR lateral score drops from 2.9 to 0.3), confirming their importance.
- Using LLMs like GPT-4o for semantic evaluation of view-specific reports surpasses traditional metrics, capturing clinical relevance. Ablation studies confirm the effectiveness of VND and VNV modules in enhancing multi-view report coherence.
Significance
This work advances understanding of neuron-level mechanisms in Transformer-based medical report generation, offering a parameter-efficient tuning strategy to address view-dependent inconsistencies. It significantly improves the reliability and accuracy of automated radiology reports, especially when some views are missing or inconsistent. The approach enhances model interpretability and provides a scalable solution for multi-modal medical AI, paving the way for more trustworthy clinical decision support systems.
Technical Contribution
Key innovations include: • A view-specific neuron detection mechanism based on activation and attribution analysis; • Perturbation-based validation to confirm neuron importance; • A parameter-efficient fine-tuning scheme that updates only target neurons, drastically reducing training overhead. The method integrates seamlessly into Transformer architectures, validated on large-scale medical datasets, demonstrating clear improvements in multi-view report quality and interpretability.
Novelty
This is the first systematic neuron-level analysis for multi-view radiology report generation, identifying view-specific neurons via attribution and activation ranking. The combined detection, validation, and selective fine-tuning approach offers a novel paradigm that balances efficiency, interpretability, and performance, addressing the core challenge of view-dependent variability—a gap in existing fusion or full-model fine-tuning methods.
Limitations
- The approach relies on accurate view labels; errors in view classification could impair neuron identification. Its effectiveness on highly abnormal or atypical cases remains to be tested.
- Currently validated only on Transformer-based models; applicability to other architectures needs further exploration.
- Extreme view variations or cases with multiple overlapping views may challenge the robustness of the neuron selection and fine-tuning process.
Future Work
Future directions include extending neuron-level analysis to multi-modal data such as CT and MRI, developing adaptive neuron selection strategies, and exploring model-agnostic fine-tuning methods. Additionally, integrating reinforcement learning to dynamically identify view-specific neurons could further improve robustness and generalization, pushing AI-assisted diagnosis toward more reliable clinical deployment.
AI Executive Summary
The rapid development of automated radiology report generation (RRG) has transformed medical imaging analysis, yet multi-view X-ray images continue to pose significant challenges. Traditional fusion-based methods often overlook the intrinsic differences between views, leading to inconsistent and sometimes contradictory reports that undermine clinical trust. Recognizing this, the current study introduces View-PNDF, a novel framework rooted in neuron-level analysis, to enhance multi-view report consistency.
At its core, View-PNDF employs three key modules: View-specific Neuron Detection (VND), which identifies neurons highly responsive to particular views through activation and attribution scores; Neuron Verification (VNV), which validates these neurons by measuring the impact of their perturbation on report quality; and Selective Fine-tuning (VNF), which updates only these view-specific neurons, drastically reducing training costs while improving view coherence. This targeted approach leverages the interpretability of neural networks, focusing on the internal components responsible for view-dependent discrepancies.
Experimental results on large-scale datasets like IU-XRay and MIMIC-CXR demonstrate that View-PNDF significantly outperforms existing fusion and fine-tuning strategies. Metrics such as BLEU-1 reaching 0.510 and ROUGE-L of 0.396, along with clinical F1 scores of 0.414, confirm the method’s ability to generate more accurate, consistent reports across multiple views. Furthermore, LLM-based semantic evaluations reveal that the approach captures clinically relevant findings better than traditional metrics, emphasizing its practical value.
This work not only advances the technical frontier in multi-view medical report generation but also enhances the interpretability and efficiency of neural network tuning. By focusing on a small subset of neurons, it offers a scalable, explainable solution adaptable to various architectures and modalities. Future efforts will extend this neuron-centric paradigm to other imaging types and multi-task scenarios, fostering more reliable AI tools for clinical practice. Despite some limitations, such as dependence on view labels and model architecture constraints, the proposed framework marks a significant step toward trustworthy, high-quality automated medical reporting.
Deep Dive
Plain Language Accessible to non-experts
想象你在厨房做饭，不同的菜谱需要用不同的厨具和食材。有些厨具专门用来做某种菜，比如炒锅用来炒菜，蒸锅用来蒸包子。每次做菜时，厨师会根据菜的不同选择不同的厨具。有时候，厨师会特别擅长用某些厨具做特定的菜，所以他会多练习那些厨具，变得更拿手。这个研究就像是在找出厨房里那些专门用来做特定菜的厨具（神经元），然后只帮他们多练习，让整个厨房（模型）变得更擅长做多种菜（报告）。这样，不管用哪个厨具做菜，菜都能做得又快又好，而且味道一致。通过这种方法，厨师们变得更专业，做出来的菜也更美味啦！
ELI14 Explained like you're 14
想象你在学校里，有很多老师教不同的科目。有的老师特别擅长数学，有的老师擅长语文。有时候，老师们讲课的方式不一样，学生会觉得很混乱。这篇文章就像是在找出那些专门教数学的老师（神经元），然后只帮他们多练习，让他们在讲数学题时更清楚、更一致。这样，不管哪个老师讲课，学生都能听懂，也不会搞错。通过这样的方法，老师们变得更擅长自己教的科目，学生也学得更好啦！
Abstract
Recent years have seen substantial advances in radiology report generation (RRG), yet existing approaches predominantly adopt direct feature fusion when handling multi-view X-ray images. Such approaches overlook the potential clinical inconsistencies and inaccuracies arising when a single model processes different views, adversely impacting performance and clinical reliability. To this end, we introduce View-PNDF (View-specific Pattern Neuron Detection and Fine-tuning), a parameter-efficient framework that fosters view-consistent report generation from a neuronal perspective. Specifically, View-PNDF comprises: (i) a view-specific neuron detection module identifying neurons responsive to particular views, (ii) a verification module quantifying the existence of these neurons, and (iii) a selective fine-tuning strategy strengthening detected neurons while preserving view-agnostic representations. By updating only view-specific neurons, View-PNDF achieves consistent diagnoses across different views with reduced computational costs. Subsequently, we employ Large Language Models (LLMs) to consolidate the view-specific reports into a complete radiology report. Furthermore, we use traditional Natural Language Generation (NLG) metrics-based assessment on integrated reports for baseline comparison and employ LLM-based assessment (e.g., GPT-4o) on view-specific reports to capture clinical significance. Extensive experiments on two medical RRG benchmarks demonstrate that View-PNDF substantially improves view-specific chest X-ray report generation quality while maintaining robust general-view performance.