KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers
KaggleDBQA leverages real-world databases and documentation, boosting Text-to-SQL accuracy by over 13.2% in practical settings.
Key Findings
Methodology
This work constructs KaggleDBQA by collecting authentic Kaggle databases with domain-specific, unnormalized structures. It gathers unrestricted natural questions from multiple annotators, paired with SQL annotations, filtering out unexpressible queries. The study integrates database documentation—column descriptions and metadata—to serve as implicit knowledge. Baseline models (RAT-SQL, EditSQL) trained on Spider are evaluated on this dataset, with additional fine-tuning and knowledge augmentation. Experiments analyze zero-shot and few-shot performance, demonstrating that incorporating documentation improves accuracy by 13.2%, nearly doubling baseline results.
Key Results
- In zero-shot evaluation, RAT-SQL achieves 13.56% accuracy on KaggleDBQA, but with document augmentation and few-shot fine-tuning, accuracy rises to 26.77%, nearly doubling the performance.
- Few-shot learning with 10-20 in-domain examples boosts accuracy significantly, with the best results obtained after combining schema descriptions and adaptation, surpassing 26%.
- Complex SQL queries, including multi-table joins and nested queries, show marked performance improvements when models leverage database documentation, validating the approach’s effectiveness.
Significance
This research addresses the gap between academic datasets and real-world industrial databases, emphasizing the importance of domain-specific knowledge and realistic data. It demonstrates that integrating database documentation and few-shot adaptation substantially enhances model robustness and transferability, crucial for deploying practical NL-to-SQL systems in diverse industries. The approach paves the way for more resilient, scalable, and user-friendly database interfaces, reducing reliance on specialized query languages and lowering deployment barriers.
Technical Contribution
The paper introduces a novel dataset, KaggleDBQA, based on authentic industrial databases, emphasizing real-world complexity. It innovates by integrating database documentation as an implicit knowledge source, employing artificial description augmentation, and proposing a few-shot learning framework. The schema linking mechanism is extended to incorporate descriptions, improving understanding of obscure column names. The evaluation framework simulates practical deployment scenarios, emphasizing knowledge transfer and adaptation, thus bridging the gap between academic benchmarks and industrial needs.
Novelty
This is the first work systematically incorporating real-world database documentation into Text-to-SQL models, demonstrating significant performance gains in industrial environments. Unlike prior datasets focused on normalized schemas, KaggleDBQA reflects true database complexity, and the combined use of few-shot learning and documentation is a novel approach that enhances transferability and robustness, setting a new standard for practical evaluation.
Limitations
- Relies heavily on manual extraction and curation of database documentation, which is costly and difficult to scale across diverse industries.
- Model performance on extremely complex or ambiguous questions remains limited, indicating the need for further understanding of implicit knowledge and context.
- Current experiments focus on specific domains; cross-industry generalization and multilingual adaptation require additional research.
Future Work
Future directions include automating database documentation extraction via NLP techniques, expanding to multilingual and multi-industry datasets, and integrating structured knowledge graphs for deeper reasoning. Developing models capable of continual learning from evolving databases and automating schema understanding will further bridge the gap between academic research and industrial deployment.
AI Executive Summary
The rapid evolution of natural language understanding (NLU) has propelled the development of Text-to-SQL systems, enabling users to query databases using natural language. However, most existing benchmarks like Spider and WikiSQL are limited to academic settings, often involving normalized schemas and curated questions, which do not reflect the complexity of real-world industrial databases. These datasets lack the obscurity, domain-specific terminology, and implicit knowledge present in actual enterprise environments, leading to a significant gap between research and deployment.
To address this, the authors introduce KaggleDBQA, a novel dataset built from authentic Kaggle databases. These databases feature unnormalized schemas, domain-specific data types, and minimal preprocessing, closely mirroring industrial data. The dataset includes unrestricted natural questions collected from multiple annotators, paired with SQL annotations, and supplemented with rich database documentation, such as column descriptions and metadata. This setup aims to simulate real-world scenarios where models must understand obscure column names and leverage implicit domain knowledge.
Experimental results demonstrate that state-of-the-art models like RAT-SQL and EditSQL perform poorly on KaggleDBQA in zero-shot settings, with accuracy around 13.56%. However, by incorporating database documentation and applying few-shot fine-tuning, accuracy nearly doubles to 26.77%. These findings highlight the importance of external knowledge sources and minimal in-domain data for practical deployment. The study also shows that models benefit significantly from schema descriptions, especially when combined with adaptation strategies, enabling better handling of complex SQL queries.
This work has broad implications for industry, suggesting that realistic evaluation frameworks and knowledge augmentation are essential for building robust, scalable NL-to-SQL systems. Future research will focus on automating documentation extraction, expanding to diverse domains, and integrating structured knowledge to further improve performance and usability in real-world applications.
Deep Analysis
Background
Database question answering (DBQA) has evolved from rule-based systems to deep learning models, with datasets like GeoQuery, ATIS, WikiSQL, and Spider driving progress. Early efforts focused on small, well-structured datasets, but these lacked the complexity of industrial databases. Recent datasets aim to evaluate cross-domain generalization, yet they often rely on normalized schemas and curated questions, limiting real-world applicability. Industrial databases typically feature obscure naming, domain-specific data types, and implicit knowledge, posing significant challenges for existing models. Addressing these gaps requires constructing datasets that reflect real operational environments, incorporating domain knowledge, and evaluating models under practical constraints. This research aligns with industry needs for scalable, robust, and adaptable NL-based database interfaces.
Core Problem
Despite advances in Text-to-SQL, deploying these models in real-world settings remains difficult. The core issues include handling complex, unnormalized schemas with obscure naming conventions, understanding diverse and unconstrained natural questions, and leveraging implicit domain knowledge such as documentation. Existing datasets do not capture these challenges, leading to models that perform well academically but poorly in production. The lack of realistic evaluation metrics and training data hampers progress. Moreover, models trained solely on normalized, curated datasets struggle with real industrial data, limiting their practical utility. Overcoming these barriers requires datasets and methods that simulate actual deployment scenarios, emphasizing knowledge transfer and robustness.
Innovation
This work introduces several key innovations:
1) Building KaggleDBQA from real Kaggle databases, capturing authentic complexity and naming conventions.
2) Incorporating database documentation, such as column descriptions, as implicit knowledge to improve schema linking.
3) Developing a few-shot learning framework, allowing models to adapt with minimal in-domain data.
4) Artificially augmenting schemas with descriptions to bridge train-test gaps.
5) Demonstrating that knowledge integration and adaptation significantly improve performance, setting a new standard for practical evaluation.
These innovations collectively address the limitations of prior datasets and models, pushing the field toward real-world readiness.
Methodology
- �� 数据采集：从Kaggle平台随机选择符合条件的数据库，确保包含未预处理的结构和丰富的文档信息。
- �� 问句采集：由多名标注员在无任务提示下，为每个数据库编写10个自然问句，确保问句多样且贴近实际。
- �� SQL标注：由专业SQL专家为每个问句标注对应SQL，筛除无法表达的问句，保留272个有效样本。
- �� 文档增强：提取数据库列描述，作为隐性知识源，人工整理并与模型输入结合。
- �� 模型训练：采用RAT-SQL，结合少样本微调和文档描述，进行多轮训练和验证。
- �� 实验设置：在零样本和少样本条件下评估模型性能，比较加入文档和不加入的效果，分析复杂SQL的表现差异。
Experiments
采用真实Kaggle数据库，划分70%为测试集，30%为训练集。基线模型为RAT-SQL和EditSQL，训练在Spider数据集上。评估指标为准确率（exact match）。实验包括：
- �� 零样本评估：模型在未见过的工业数据库上测试，验证迁移能力。
- �� 少样本微调：在每个数据库的少量样本基础上微调，观察性能提升。
- �� 文档增强：引入列描述信息，分析对模型理解和SQL生成的影响。
- �� 复杂SQL分析：统计不同难度等级的SQL结构，评估模型在复杂查询中的表现。
- �� 跨行业迁移：采用留一法，验证模型在不同数据库间的泛化能力。
Results
模型在KaggleDBQA上的准确率显著低于在Spider上的表现，零样本准确率为13.56%。引入少样本微调后，准确率提升至26.77%，增长超过13个百分点。结合列描述信息，模型性能进一步提升，尤其在复杂SQL和模糊问句中表现优异。实验还显示，模型对工业数据库中的命名模糊和结构复杂性具有一定适应能力，验证了知识增强策略的有效性。这些结果表明，结合真实数据和隐性知识是提升工业环境中Text-to-SQL性能的关键。
Applications
该技术可应用于企业内部数据分析、智能客服、自动报表生成等场景。企业数据库通常结构复杂、命名模糊，模型需结合文档信息进行迁移。实现条件包括：丰富的数据库文档、少样本标注能力和鲁棒的模型架构。未来，结合自动化文档提取和知识图谱，将大幅降低部署成本，提升系统智能化水平。
Limitations & Outlook
模型在极端复杂SQL或极度模糊问句中仍表现不足，依赖人工整理的数据库文档成本较高。当前实验主要集中在特定行业数据库，跨行业迁移效果有待验证。模型对极端命名或少量样本的适应性仍需提升，未来需结合自动化知识提取和多模态信息融合以增强鲁棒性。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有许多机器（数据库），每台机器都有标签（列名）告诉你它的功能，但这些标签都很难懂，甚至是缩写。你需要找到某个产品（问句）对应的机器，但标签不直观。工厂的手册（数据库文档）里写了每台机器的详细介绍，帮你理解机器的作用。通过阅读手册，你可以更快找到正确的机器，即使标签难懂。这个过程就像模型利用数据库文档，理解复杂的数据库结构，从而更准确地回答自然语言问题。没有手册时，找机器就像盲人摸象，容易出错；有手册后，效率大大提高。这个比喻说明，结合隐性知识能让数据库问答变得更智能、更实用。
ELI14 Explained like you're 14
想象你在学校图书馆找书，但书架上的标签都用奇怪的缩写，难以理解。你想问老师：“哪个书架有关于足球的书？”但标签不直观，难以找到。于是老师给你一本说明书，详细介绍每个标签代表的内容。你看完说明书后，就能更快找到足球书了。这就像研究中用数据库文档帮助模型理解数据库结构，让它更聪明地回答你的问题。没有说明书时，模型就像盲人摸象，容易出错；有了说明书，模型就像有了指南针，能更准确地找到答案。这种结合知识的方法，让数据库问答变得更贴近实际生活，也更实用。
Glossary
Text-to-SQL (文本到SQL)
将自然语言问题转换为SQL查询的技术，帮助用户用自然语言操作数据库。
本文的核心任务是提升此技术在工业环境中的表现。
Zero-shot learning (零样本学习)
模型在未见过目标任务或数据的情况下，直接进行预测的能力。
评估模型在未见过数据库上的迁移能力。
Few-shot learning (少样本学习)
模型通过少量示例快速适应新任务的能力，减少对大量标注数据的依赖。
本文采用少样本微调提升模型性能。
Database documentation (数据库文档)
描述数据库结构、字段含义和使用方法的文本资料，提供隐性知识。
用于增强模型理解复杂数据库的能力。
Schema linking (模式连接)
将自然语言中的实体或属性与数据库中的结构元素对应的过程。
模型在理解问句时的关键步骤。
Open Questions Unanswered questions from this research
- 1 如何自动化提取和更新数据库文档以支持模型持续学习？
- 2 在极端复杂或模糊问句中，模型的理解能力如何进一步提升？
- 3 跨行业迁移中，模型对不同数据库命名和结构差异的适应性如何增强？
Applications
Immediate Applications
企业数据分析
企业可以利用该技术实现自然语言查询，快速获取财务、销售等关键数据，降低技术门槛。
智能客服与自动报告
结合数据库文档，提升客服系统的智能化水平，实现自动生成业务报告。
Long-term Vision
普及工业级自然语言数据库问答
未来可实现全行业普遍部署，降低数据库操作门槛，推动智能化转型。
Abstract
The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.