INFOTABS: Inference on Tables as Semi-structured Data

The evolution of NLP has seen significant advances with models like BERT and RoBERTa, which excel at understanding and reasoning over unstructured text. However, real-world information often exists in semi-structured formats such as tables, info-boxes, and spreadsheets, which pose unique challenges due to their heterogeneity and implicit relationships. Prior datasets like TabFact and approaches such as table-based attention models have made progress but are limited in scope, often focusing on simple fact verification or structured data. The complexity of reasoning involving multiple rows, temporal data, numerical calculations, and commonsense knowledge remains underexplored. As AI systems are increasingly deployed in practical scenarios—medical diagnosis, financial analysis, scientific research—the ability to interpret semi-structured data accurately becomes critical. This background underscores the necessity for datasets and models capable of handling the multifaceted reasoning required for real-world applications.

Despite advances, current models struggle with reasoning over semi-structured data, especially when multiple inference steps, domain shifts, or external knowledge are involved. The core challenge lies in the heterogeneity and implicit relationships within tables, which demand multi-faceted reasoning beyond surface pattern matching. Existing datasets often lack the complexity needed to evaluate these capabilities, leading to overfitting and superficial understanding. Consequently, models perform well on training-like data but poorly generalize to real-world scenarios requiring multi-step, cross-domain, and commonsense reasoning. Addressing this bottleneck requires constructing challenging datasets, developing representations that capture structural nuances, and designing models capable of multi-faceted inference.

The paper's key innovations include: 1) INFOTABS dataset, comprising diverse reasoning types over Wikipedia info-box tables, with multiple challenge test sets to evaluate generalization; 2) multiple table representations—paragraph, key-value, attention-based—to explore different ways of encoding semi-structured data; 3) comprehensive analysis of reasoning categories, emphasizing multi-step, temporal, numerical, and commonsense reasoning. The dataset's design ensures high-quality annotations validated via crowdsourcing, and the test sets control for biases, pushing models beyond superficial pattern recognition. The integration of diverse reasoning types and representations marks a significant step forward in semi-structured data understanding.

• �� Data collection: Extract tables from Wikipedia info-boxes, filter for size and content, simplify into key-value pairs. • Sentence generation: Crowdsourcing workers create three sentences per table—entailment, contradiction, neutral—ensuring coverage of reasoning types. • Data validation: Multiple annotators verify labels, achieving high inter-annotator agreement (Kappa ~0.8). • Table representations: Convert tables into paragraphs, structured key-value sequences, and attention-based encodings. • Model training: Use pre-trained models (BERT, RoBERTa) with classification heads, trained on different representations. • Challenge sets: Construct α1 (similar distribution), α2 (adversarial perturbations), α3 (cross-domain) to evaluate robustness. • Reasoning analysis: Categorize reasoning types (multi-row, temporal, KCS), analyze model performance across categories.

Models trained include SVM with n-grams, BERT/BERT-based classifiers, RoBERTa variants, with various input representations. Hyperparameters tuned via validation, using Adam or Adagrad optimizers, learning rates around 10^-4. The dataset is split into training, validation, and three challenge test sets. Experiments assess in-domain performance, hypothesis bias, and cross-domain generalization. Ablation studies compare different table encodings, and bias detection experiments evaluate reliance on superficial cues. Results are averaged over multiple runs to ensure robustness. The evaluation metrics include accuracy, F1-score, and human agreement levels, providing a comprehensive picture of model capabilities.

Models perform well on α1 (~66%) but drop below 50% on α2 and α3, indicating difficulty in generalizing to adversarial and cross-domain data. Structured encodings and multiple representations improve performance modestly but do not close the gap with human accuracy (~80%). Analysis of reasoning categories shows many instances require multi-step, temporal, or commonsense reasoning, which current models handle poorly. Human annotators exhibit high agreement, confirming data quality. The results highlight the persistent challenge of modeling complex reasoning over semi-structured data, emphasizing the need for new architectures and external knowledge integration.

This dataset and modeling framework can be applied to improve question answering systems, automated fact-checking, and intelligent information retrieval where structured data is prevalent. It can also serve as a benchmark for developing models that better understand complex relationships in semi-structured formats, impacting industries like healthcare, finance, and scientific research. The insights gained can guide the design of more robust, interpretable AI systems capable of reasoning akin to human cognition, especially in multi-source, multi-step inference tasks.

Despite advancements, models still underperform on multi-step, multi-domain reasoning, especially in cross-domain scenarios. The dataset, while diverse, cannot encompass all possible reasoning types, and models lack transparency in reasoning processes. High computational costs and reliance on pretraining limit scalability. Moreover, external knowledge integration remains limited, constraining reasoning depth. Future work must address these issues by developing more explainable, knowledge-aware models and expanding datasets to cover broader reasoning scenarios.