TQA-Bench: Evaluating LLMs for Multi-Table Question Answering
TQA-Bench evaluates LLMs in multi-table QA with context lengths from 8K to 64K tokens.
Key Findings
Methodology
TQA-Bench employs a flexible sampling mechanism supporting context lengths from 8K to 64K tokens, combined with symbolic extensions to assess LLMs' reasoning capabilities in multi-table scenarios. Systematic evaluation of LLMs ranging from 2 billion to 671 billion parameters reveals their performance in complex data environments.
Key Results
- LLMs perform better in single-table scenarios, with up to 20% higher accuracy than multi-table.
- Markdown serialization outperforms CSV, JSON, and HTML formats.
- Instruction-tuned LLMs excel in long-context settings, especially in complex reasoning tasks.
Significance
This study bridges the gap between existing single-table QA benchmarks and practical application needs, advancing LLMs' application in complex data management tasks, particularly in finance, healthcare, and e-commerce.
Technical Contribution
TQA-Bench introduces variable context lengths and symbolic extensions, providing a reliable multi-table QA evaluation framework that surpasses simple data retrieval and pattern matching.
Novelty
First systematic evaluation of LLMs in multi-table QA, particularly in long-context and complex reasoning tasks.
Limitations
- LLMs still struggle with complex analytical queries, especially in generating semantically correct SQL.
- Current models show significant performance degradation in long contexts.
- Symbolic extension methods may lead to insufficient diversity in evaluation results.
Future Work
Future research could explore improving LLMs' performance in complex analytical tasks and developing more effective context management strategies.
AI Executive Summary
The advancement of large language models (LLMs) has unlocked significant opportunities in complex multi-modal data management tasks, particularly in question answering (QA) over complicated multi-table relational data. However, systematically evaluating LLMs on multi-table QA remains a critical challenge due to the inherent complexity of analyzing the modality of relational data structures and the potentially large scale of serialized tabular data. Existing benchmarks primarily focus on single-table QA, failing to capture the intricacies of connections across multiple relational tables, as required in real-world domains such as finance, healthcare, and e-commerce.
We present TQA-Bench, a long-context analytical multi-table QA benchmark derived from real-world public datasets, with a flexible sampling mechanism that varies context length (8K-64K tokens) and symbolic extensions for assessing reasoning beyond retrieval and pattern matching. We systematically evaluate a set of LLMs spanning model scales from 2 billion to 671 billion parameters. Our extensive experiments reveal critical insights into the performance of LLMs in multi-table QA, highlighting both challenges and opportunities for advancing their application in complex, data-driven environments.
TQA-Bench introduces variable context lengths and symbolic extensions, providing a reliable multi-table QA evaluation framework that surpasses simple data retrieval and pattern matching. This study bridges the gap between existing single-table QA benchmarks and practical application needs, advancing LLMs' application in complex data management tasks, particularly in finance, healthcare, and e-commerce.
Deep Dive
Abstract
The advance of large language models (LLMs) has unlocked great opportunities in complex multi-modal data management tasks, particularly in question answering (QA) over complicated multi-table relational data. Despite significant progress, systematically evaluating LLMs on multi-table QA remains a critical challenge due to the inherent complexity of analyzing the modality of relational data structures and the potentially large scale of serialized tabular data. Existing benchmarks primarily focus on single-table QA, failing to capture the intricacies of connections across multiple relational tables, as required in real-world domains such as finance, healthcare, and e-commerce. We present TQA-Bench, a long-context analytical multi-table QA benchmark derived from real-world public datasets, with a flexible sampling mechanism that varies context length (8K--64K tokens) and symbolic extensions for assessing reasoning beyond retrieval and pattern matching. We systematically evaluate a set of LLMs spanning model scales from 2 billion to 671 billion parameters. Our extensive experiments reveal critical insights into the performance of LLMs in multi-table QA, highlighting both challenges and opportunities for advancing their application in complex, data-driven environments.