TriQua: Reconciling Granularity and Context in Factuality Evaluation
Proposes TriQua framework, combining base triples with qualifiers for fine-grained factuality evaluation, achieving high correlation (r=0.89) with human scores.
Key Findings
Methodology
TriQua employs an open information extraction approach, representing simple facts as (subject, relation, object) triples, and complex facts as triples augmented with independent qualifiers. The framework integrates a decomposer, retriever, and verifier, with the verifier leveraging large language models (e.g., Qwen3.5-397B) to perform fact verification via natural language inference. The verification process annotates specific errors within triples and qualifiers, enabling fine-grained explainability. The core scoring metric, TriQuaScore, calculates the ratio of supported fact units, incorporating deduplication to improve robustness. This structure supports multi-layered fact analysis, balancing atomicity and contextual richness, suitable for long-form text and complex assertions.
Key Results
- On FactScore-Bio, TriQuaScore correlates with human judgments at 0.89, outperforming VeriScore and RefChecker, with a 4-10% reduction in mean absolute error. In complex scenarios, entailment support reaches 98.54%, and decontextualization support reaches 95.39%. Ablation studies confirm the importance of relation-based scoring and deduplication strategies, enhancing robustness and interpretability.
- TriQua demonstrates superior decomposition quality, with high entailment and decontextualization rates across datasets. Its ability to support partial scoring and error localization enables more nuanced evaluation, especially in long, complex texts. The framework maintains high performance across multiple model scales, including Qwen3.5 and GLM-744B, showing scalability and consistency.
- Experimental results validate that the structured hyperrelational representation effectively captures complex facts, and the relation-based metric provides detailed support/error attribution. The approach significantly reduces false negatives and improves interpretability, making it suitable for real-world applications like knowledge base curation and automated fact-checking.
Significance
This work advances the field of factuality evaluation by bridging the gap between atomic fact decomposition and contextual understanding. Its structured representation and relation-centric scoring enable models to handle complex, nuanced assertions with explainability, addressing longstanding challenges in AI trustworthiness. The methodology enhances both the accuracy and interpretability of fact verification, fostering more reliable AI systems in critical applications like medical diagnosis, legal analysis, and information retrieval. It also opens pathways for integrating structured knowledge into large language models, promoting more transparent AI reasoning.
Technical Contribution
The paper introduces a hyperrelational fact representation framework that extends traditional triples with independent qualifiers, enabling nuanced expression of complex facts. It develops the TriQuaScore metric, which decomposes facts into supportable units and supports partial credit, improving evaluation granularity. The approach combines open information extraction with few-shot learning, avoiding reliance on fixed ontologies. Empirical validation across multiple datasets and model sizes demonstrates its robustness, scalability, and superior alignment with human judgments, setting a new standard for factuality assessment.
Novelty
This is the first work to systematically incorporate hyperrelational structures into factuality evaluation for unstructured text, effectively balancing atomicity and contextual richness. Unlike prior methods limited to simple triples or sentence-level analysis, TriQua’s multi-dimensional qualifiers allow detailed contextualization without sacrificing interpretability. Its relation-based scoring mechanism offers a novel, explainable metric that supports partial correctness, setting a new paradigm in factuality assessment and interpretability in NLP.
Limitations
- The framework relies heavily on large pre-trained models, which entail high computational costs, limiting real-time deployment in resource-constrained environments.
- Qualifiers are manually categorized, and their automatic classification remains challenging, potentially leading to misinterpretation or over-embedding of context.
- Handling highly abstract or ambiguous facts still poses difficulties, especially when multiple interpretations exist or when external knowledge sources are incomplete.
Future Work
Future research will focus on integrating multimodal data (images, videos) for comprehensive fact validation, automating qualifier classification, and reducing computational overhead. Additionally, expanding the framework to support causal and temporal reasoning, and exploring reinforcement learning to improve fact extraction and verification accuracy, are promising directions. The goal is to develop more autonomous, scalable, and explainable fact-checking systems suitable for deployment in real-world, high-stakes scenarios.
AI Executive Summary
The rapid deployment of large language models (LLMs) in diverse applications has underscored the critical importance of factual accuracy. Existing approaches to factuality evaluation, such as the 'decompose-then-verify' paradigm, have shown promise in breaking down complex statements into manageable units. However, these methods often face a fundamental trade-off: simple atomic facts lack sufficient context for precise verification, while broader claims sacrifice granularity, leading to potential misjudgments. Addressing this challenge, the authors introduce TriQua, a novel framework that models facts with a flexible structure combining base triples and independent qualifiers. This hyperrelational representation captures both the core information and necessary contextual modifiers, such as temporal, spatial, or causal details, without losing atomicity. The framework employs a relation-based scoring metric, TriQuaScore, which evaluates the support for each fact unit, allowing partial credit and detailed error attribution. Experimental results on datasets like FactScore-Bio demonstrate that TriQua achieves a high correlation (r=0.89) with human judgments, outperforming existing methods like VeriScore and RefChecker. Its decomposition quality remains robust across complex and abstract claims, with high entailment and decontextualization support rates. The system's ability to precisely locate errors enhances interpretability, making it suitable for critical applications like knowledge base curation and automated fact-checking. The authors also validate the effectiveness of deduplication strategies and multi-model validation, ensuring reliability across diverse scenarios. Looking ahead, future work aims to incorporate multimodal data, automate qualifier classification, and optimize computational efficiency, paving the way for more autonomous and trustworthy AI systems. Overall, TriQua represents a significant step forward in balancing factual granularity and contextual richness, fostering more transparent and reliable AI reasoning in complex real-world tasks.
Deep Dive
Abstract
The "decompose-then-verify" paradigm for LLM factuality evaluation faces a fundamental trade-off: atomic facts, i.e., one sentence conveying one unit of information, often omit essential context, while broader statements lack the granularity needed for precise assessment. To address this, we introduce TriQua, a framework that flexibly models facts based on their complexity. Simple claims are extracted as standard triples, while complex claims are represented as hyperrelational facts by attaching auxiliary contextual qualifiers. This adaptive structure preserves the necessary context for accurate retrieval and verification without sacrificing atomicity. Furthermore, TriQua's verification process directly annotates concrete errors within specific triples and qualifiers, providing fine-grained explainability for error detection. Alongside the framework, we propose TriQuaScore to quantify the factuality of these structured fact units. Empirical evaluations show that TriQuaScore strongly aligns with human annotated factuality scores, TriQua achieves robust decomposition quality, and outperforms existing decomposition-based frameworks in evidence-based fact verification.