CRAFT: A Unified Counterfactual Reasoning Framework for Tabular Question Answering and Fact Verification
CRAFT introduces bidirectional counterfactual reasoning, achieving 82.4% accuracy on WikiTQ and 94.6% on TabFact, outperforming baselines.
Key Findings
Methodology
CRAFT comprises four modules: Rewriter converts questions into declarative statements; Reverser generates counterfactual statements; Extractor retrieves evidence from both paths; Rethinker fuses evidence and answers. It explicitly constructs dual reasoning paths, utilizing weighted evidence integration. Techniques include type-consistent sampling, SQL-style verification program generation, and multi-path evidence aggregation. The framework is applicable to datasets like WikiTQ and TabFact, enhancing robustness and interpretability in structured data reasoning.
Key Results
- On WikiTQ and TabFact, CRAFT achieves 82.4% and 94.6% accuracy, respectively, surpassing the best baselines by 4.7 and 1.1 percentage points, especially excelling in complex question answering scenarios.
- Across models like LLaMA 3.3B and Qwen 2.5B, performance gaps narrow significantly, demonstrating the method's robustness and generalizability.
- Ablation studies reveal that combining Rewriter and Reverser yields better evidence coverage and reasoning diversity than single-path approaches.
Significance
This work addresses the limitations of unidirectional reasoning in large language models, introducing a structured, interpretable, and robust paradigm. By explicitly modeling counterfactual paths, it enhances multi-step reasoning over long, structured tables, advancing both theoretical understanding and practical capabilities in data comprehension. The approach improves accuracy, stability, and cross-model performance, offering a new foundation for trustworthy structured reasoning systems with broad implications for NLP tasks involving complex data structures.
Technical Contribution
The paper proposes a novel counterfactual reasoning framework that integrates multiple reasoning paths with evidence fusion, providing theoretical guarantees for information maximization. It innovates by transforming questions into declarative statements, generating counterfactual variants, and employing multi-path evidence aggregation, setting a new standard for robustness and interpretability in table reasoning. This approach extends the capabilities of existing models without requiring extensive parameter tuning or external tools.
Novelty
This is the first systematic integration of counterfactual reasoning into structured table inference tasks. Unlike prior work limited to unidirectional inference or external symbolic engines, CRAFT explicitly constructs and leverages counterfactual paths, enriching the reasoning process with alternative hypotheses and evidence contrast, thus offering a fundamentally new perspective on structured data understanding.
Limitations
- The framework's reliance on multiple reasoning paths increases computational overhead, which may limit real-time applications.
- Counterfactual generation depends on templates and rules, potentially reducing flexibility in highly complex or noisy data scenarios.
- Handling extremely long tables or multi-relational data remains challenging, requiring further integration with external knowledge bases or reasoning engines.
Future Work
Future research will focus on optimizing path generation efficiency, incorporating external knowledge for enhanced reasoning, extending to multi-modal structured data, and developing theoretical guarantees for evidence fusion. Exploring scalable, real-time implementations and broader domain applications will also be key directions.
AI Executive Summary
Structured data in the form of tables plays a crucial role across many fields, from finance to scientific research. Despite the impressive capabilities of large language models (LLMs), their ability to perform multi-step, long-range reasoning over complex tables remains limited. Existing approaches often rely on unidirectional inference, which constrains the exploration of alternative hypotheses and reduces robustness. To address these challenges, this paper introduces CRAFT, a unified counterfactual reasoning framework that explicitly constructs both original and counterfactual reasoning paths.
CRAFT's core innovation lies in transforming questions into declarative statements, generating counterfactual variants, and then extracting evidence from both paths. These evidences are integrated via a weighted mechanism, enabling the model to consider multiple reasoning trajectories simultaneously. This multi-path approach not only enhances reasoning depth but also improves the model's ability to reject spurious correlations and ambiguous evidence.
Experimental results on datasets like WikiTQ and TabFact demonstrate the effectiveness of CRAFT. The framework achieves 82.4% accuracy on WikiTQ and 94.6% on TabFact, outperforming existing baselines by significant margins. Notably, the improvements are most pronounced in complex question answering scenarios, where traditional models often struggle. The robustness of the approach is validated across different backbone models, including LLaMA and Qwen, with performance gaps narrowing substantially.
The significance of this work extends beyond immediate performance gains. By explicitly modeling counterfactual paths, CRAFT offers a more interpretable and reliable reasoning paradigm, paving the way for trustworthy AI systems capable of nuanced understanding of structured data. It addresses fundamental limitations of current models and opens new avenues for research into multi-path, multi-hypothesis reasoning frameworks.
Looking ahead, future work will explore more efficient path generation, integration with external knowledge bases, and extension to multi-modal structured data. The ultimate goal is to develop scalable, explainable, and highly accurate reasoning systems that can operate reliably in real-world applications, from automated report analysis to intelligent decision support systems.
Deep Dive
Abstract
Table reasoning remains challenging for large language models (LLMs), particularly in tasks that require multi-step inference over long and structured tables. Existing approaches predominantly rely on single-direction reasoning, which limits their ability to explore alternative hypotheses across tasks. In this work, we propose CRAFT, a unified Counterfactual Reasoning Framework that reformulates Tabular question answering and fact verification into a general bidirectional verification process. Our method explicitly constructs both declarative statements and their counterfactual variants. Evidence is then extracted from reasoning along both the original and counterfactual paths, and integrated via a weighted mechanism to arrive at the final answer. Experimental results show that our approach consistently surpasses representative baselines on table reasoning datasets such as WikiTQ and TabFact, achieving especially large improvements on complex question answering. Our framework also significantly mitigates performance gaps between different backbone LLMs. This indicates that counterfactual reasoning effectively overcomes the limitations of single-direction inference, guiding LLMs toward more discerning reasoning and establishing a more principled paradigm for structured reasoning tasks. Our code will be made publicly available upon acceptance.