ToTTo: A Controlled Table-To-Text Generation Dataset
Introduced ToTTo dataset with 120K samples, combining manual revision and highlighted cells for high-precision table-to-text generation.
Key Findings
Methodology
The study constructs a large-scale dataset by extracting Wikipedia tables and associated sentences, then employs a multi-stage annotation process where annotators highlight relevant cells, delete unsupported phrases, decontextualize references, and correct grammar. This iterative revision ensures the target sentences are both natural and faithful to the source table. The dataset supports training models like BERT-Seq2Seq and Pointer-Generator, which are evaluated on BLEU-4 and content fidelity metrics. The approach emphasizes content support, diversity, and high precision, addressing the hallucination problem prevalent in neural generation.
Key Results
- Models trained on ToTTo achieve an average BLEU-4 score of 68.98, but still exhibit hallucinations, indicating room for improvement in content fidelity.
- The dataset's complexity and diversity reveal that current models struggle with content support, especially in multi-domain and complex table scenarios.
- The annotation process's multi-stage revisions significantly improve the quality of target sentences, demonstrating the effectiveness of combining human editing with structured guidance.
Significance
This dataset advances the field by providing a high-quality benchmark for high-precision, controllable table-to-text generation, crucial for applications like automated report generation, question answering, and knowledge extraction. It addresses key issues of content accuracy and naturalness, which are vital for deploying reliable NLP systems in real-world settings. By focusing on faithful content support, ToTTo pushes the development of models that can reason over structured data and produce trustworthy narratives.
Technical Contribution
The core innovation lies in the annotation methodology that integrates human editing with highlighted cell guidance, producing high-quality, faithful data. The dataset's scale and diversity enable training more robust models. Additionally, the evaluation of multiple neural architectures highlights the persistent challenge of hallucination, guiding future research toward better content control mechanisms.
Novelty
This work is the first to systematically combine manual revision with explicit cell highlighting to ensure content fidelity in a large, diverse dataset. Unlike prior datasets relying solely on natural text or synthetic data, ToTTo emphasizes structured content support, setting a new standard for controllable data-to-text generation.
Limitations
- Despite its focus on content fidelity, models trained on ToTTo still hallucinate unsupported information, indicating the need for better reasoning and knowledge integration.
- The annotation process is labor-intensive and costly, limiting scalability for larger or multilingual datasets.
- Handling highly complex or domain-specific tables remains challenging, requiring further model and dataset enhancements.
Future Work
Future directions include automating parts of the annotation process, integrating external knowledge bases for improved reasoning, and extending the dataset to multilingual and multimodal formats. Developing models that better leverage highlighted cells and support reasoning over complex data will further enhance content fidelity and controllability.
AI Executive Summary
In recent years, neural models for data-to-text generation have made significant progress, yet they often suffer from hallucinations—producing fluent but factually unsupported content. Existing datasets either rely on noisy natural language or synthetic data, limiting the models' ability to generate faithful descriptions of structured data. To address this, the authors introduce ToTTo, a large-scale, high-quality dataset comprising over 120,000 examples, designed to facilitate high-precision, controlled table-to-text generation.
The core innovation of ToTTo lies in its annotation methodology. Annotators start with candidate sentences from Wikipedia, then iteratively revise them by highlighting relevant table cells, deleting unsupported phrases, decontextualizing references, and correcting grammar. This process ensures the target sentences are both natural and directly supported by the source table, providing a strong supervision signal for models. The dataset spans diverse domains and table structures, emphasizing content fidelity and diversity.
Experimental results using models like BERT-Seq2Seq and Pointer-Generator demonstrate that while models can generate fluent sentences, hallucination remains a challenge. The dataset exposes these weaknesses, guiding future research toward models with better content control and reasoning capabilities. Overall, ToTTo offers a valuable benchmark for advancing controllable, faithful data-to-text generation, with broad implications for automated reporting, question answering, and knowledge extraction.
Looking ahead, integrating external knowledge, automating annotation, and expanding to multilingual contexts are promising directions. The dataset's emphasis on human-guided revision and explicit content support sets a new standard for high-precision natural language generation from structured data, promising to bridge the gap between structured information and natural language understanding.
Deep Dive
Abstract
We present ToTTo, an open-domain English table-to-text dataset with over 120,000 training examples that proposes a controlled generation task: given a Wikipedia table and a set of highlighted table cells, produce a one-sentence description. To obtain generated targets that are natural but also faithful to the source table, we introduce a dataset construction process where annotators directly revise existing candidate sentences from Wikipedia. We present systematic analyses of our dataset and annotation process as well as results achieved by several state-of-the-art baselines. While usually fluent, existing methods often hallucinate phrases that are not supported by the table, suggesting that this dataset can serve as a useful research benchmark for high-precision conditional text generation.