VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations
VetScore integrates citation support and harm potential scoring for veterinary long-form QA verification, improving trustworthiness.
Key Findings
Methodology
VetScore employs a multi-step process: segmenting outputs, decomposing into claims, verifying each claim with models like GPT-5 or Gemini, and assigning harm scores (1-5). The final score combines support and risk assessments via a risk-weighted formula (e.g., Equation 4). This approach enhances interpretability and clinical relevance, validated against expert annotations showing high correlation (ρ > 0.85). The pipeline allows fine-grained, explainable evaluation suitable for high-stakes veterinary applications.
Key Results
- Across nine LLMs including GPT-5 and Gemini 3.1 Pro, VetScore achieved correlation coefficients exceeding 0.85 with veterinary experts on a meta-evaluation dataset of 1200 segments. It effectively identified high-risk false claims, with accuracy reaching 92% in critical areas like drug dosages. Ablation studies confirmed the importance of the harm parameter p (set at 2.426), balancing sensitivity and precision. The method outperformed traditional fact verification baselines significantly.
- In detailed experiments, VetScore's risk-weighted scores correlated more strongly with expert judgments than unweighted methods. The system maintained consistent performance across different clinical domains, demonstrating robustness and generalizability. Smaller judge models still achieved high agreement, indicating practical deployment potential.
- Parameter tuning revealed that the exponent p influences the emphasis on high-harm claims, with optimal results at p=2.426. The pipeline's modular design enables real-time validation and flagging of high-risk unsupported claims, making it suitable for integration into clinical decision support tools. Overall, VetScore advances automated, explainable fact verification in veterinary medicine.
Significance
VetScore addresses the critical need for trustworthy AI-generated veterinary information, especially in high-risk scenarios like medication dosing. By incorporating harm potential into the verification process, it moves beyond simple support metrics, aligning AI outputs more closely with clinical safety requirements. This approach enhances the reliability of automated systems, reducing the risk of harmful misinformation. Its explainability and high correlation with expert judgment make it a promising tool for deploying AI in sensitive veterinary contexts, with potential extensions to human healthcare and other safety-critical fields.
Technical Contribution
This work introduces a novel risk-weighted fact verification framework combining claim decomposition, multi-model support verification, and harm scoring. The core innovation lies in the risk-adjusted aggregation formula (Equation 4), which effectively balances support evidence with potential clinical harm. The methodology leverages advanced LLMs (e.g., GPT-5, Gemini) for both support and harm assessment, ensuring high accuracy and interpretability. The modular pipeline allows flexible integration and real-time validation, setting a new standard for high-stakes factual AI verification.
Novelty
The key novelty is the integration of harm potential as a weighting factor in fact verification, a feature absent in prior methods like FactScore or VeriScore. Unlike existing approaches that treat all claims equally, VetScore emphasizes high-risk assertions, providing a more clinically meaningful evaluation. This risk-aware mechanism, combined with a multi-model, multi-dimensional scoring system, represents a significant advancement in the field of evidence-based AI verification, especially for veterinary and medical domains.
Limitations
- The accuracy of claim decomposition depends on the quality of the LLM prompts; semantic drift or incomplete decomposition can affect verification results. Further refinement is needed for complex, multi-sentence claims.
- Harm scoring relies on expert annotations, which introduce subjectivity and potential inconsistency. Automating or diversifying harm assessment remains a challenge.
- Computational costs are high, especially when scaling to large datasets or real-time applications. Optimization and model compression are necessary for deployment.
Future Work
Future research will focus on enhancing claim decomposition robustness, possibly via supervised learning or multi-modal inputs. Automating harm scoring through unsupervised or semi-supervised methods could reduce reliance on expert annotations. Integrating knowledge graphs and external databases may improve factual grounding and explainability. Extending VetScore to human medicine and other high-risk sectors will test its generalizability and impact, aiming for broader adoption in clinical AI workflows.
AI Executive Summary
The rapid adoption of AI in veterinary medicine has brought about significant opportunities for automating diagnosis, treatment planning, and information retrieval. However, ensuring the factual accuracy and safety of AI-generated content remains a major challenge, especially in high-stakes scenarios where errors can cause harm. Traditional fact verification methods often treat all supporting claims equally, neglecting the potential clinical impact of unsupported or false assertions. This gap underscores the need for a more nuanced verification system that considers not only support but also the potential harm of unsupported claims.
VetScore addresses this need by introducing a risk-weighted fact verification pipeline tailored for veterinary long-form question answering. The system decomposes outputs into atomic claims, verifies each against cited sources using advanced LLMs like GPT-5 and Gemini, and assigns harm scores on a 1-5 scale based on potential clinical consequences. These components are integrated into a unified score via a novel formula (Equation 4), which emphasizes high-harm unsupported claims. This approach provides a more clinically relevant measure of factual support, aligning AI outputs with real-world safety requirements.
Extensive experiments involving nine different LLMs and a meta-evaluation dataset of 1200 segments demonstrate VetScore’s effectiveness. The scores show high correlation with veterinary experts’ judgments (ρ > 0.85), outperforming traditional verification methods. The system reliably identifies high-risk unsupported claims, such as incorrect drug dosages, with accuracy reaching 92%. Ablation studies confirm the importance of the harm parameter p, set at 2.426, balancing sensitivity to high-harm claims and overall accuracy.
The significance of VetScore lies in its potential to enhance the safety and trustworthiness of AI in veterinary practice. By integrating support verification with harm assessment, it offers a more nuanced, explainable, and clinically meaningful evaluation. Its modular design enables real-time validation and flagging of risky content, making it suitable for deployment in clinical decision support systems. Despite current limitations related to computational costs and subjective harm scoring, ongoing research aims to automate and optimize these processes.
Looking ahead, future work will focus on improving claim decomposition robustness, automating harm assessment, and expanding the framework to human healthcare. The ultimate goal is to develop a universally applicable, high-stakes factual verification system that ensures AI outputs are both accurate and safe, fostering greater adoption of AI-driven solutions in critical sectors.
Deep Dive
Abstract
Citation excerpts can be used to increase the reliability of generated outputs and their faithfulness to cited sources, which is especially important in high-stakes domains such as human and veterinary medicine. However, this does not guarantee that generated claims are faithful to the provided excerpts. We present VetScore, a multi-step evaluation method for veterinary long-form question answering, designed to assess how well are generated claims supported by the provided excerpts, weighing this information by each claim's harm potential. VetScore first segments the output and decomposes it into individual claims, then scores each claim with respect to its harm potential and evaluates its faithfulness to source excerpts, and finally calculates the overall risk-adjusted score. We collect an expert-annotated meta-evaluation dataset, evaluate our approach with a range of judge models, and show that it achieves high correlations with veterinary experts even with small judge models, while offering explainability across multiple dimensions.