A Cost-Aware Agentic Architecture for NL-to-SQL over Nested Enterprise Schemas, with a New Benchmark
Proposed a cost-aware single-generation architecture achieving 91.7% accuracy on the DevRev benchmark.
Key Findings
Methodology
The study proposes a cost-aware single-generation architecture tailored for enterprise-level nested schemas. The architecture includes schema selection, metadata retrieval, and error repair components, leveraging LLM-driven iterative schema discovery and structured error models to optimize SQL generation.
Key Results
- On the DevRev benchmark, the system achieved 91.7% answer correctness, 54.6 percentage points above the next-best baseline.
- On the Spider 2.0 Snowflake public dataset, the system is competitive at a single-generation operating point.
- Dynamic error taxonomy and history-aware feedback significantly improved the accuracy of generated SQL.
Significance
This study significantly advances the capability of natural language to SQL conversion, especially in handling complex enterprise-level nested schemas. The progress is crucial for both academic research and practical applications, addressing long-standing challenges in data querying.
Technical Contribution
Technical contributions include a new error classification and repair mechanism, and a dynamic learning cheatsheet framework. These innovations enable the system to maintain efficiency and accuracy in single-generation scenarios.
Novelty
This is the first NL2SQL architecture specifically targeting enterprise-level nested schemas, capable of handling complex multi-layer nesting and polymorphic link relationships, offering significant innovation over existing methods.
Limitations
- The system may still face performance bottlenecks when dealing with extremely complex nested schemas.
- Further optimization is needed to reduce computational costs.
Future Work
Future work includes extending the architecture to support more database types and further optimizing the error repair mechanism to enhance system robustness and efficiency.
AI Executive Summary
Natural language to SQL conversion systems have made rapid progress on academic benchmarks, but challenges remain in real-world applications, especially when dealing with enterprise-level nested schemas. Existing benchmarks fail to adequately measure these complex structures, prompting the introduction of the DevRev NL2SQL benchmark, which includes 900 execution-verified queries and introduces the Semantic Depth Score (SDS) to assess analytical reasoning depth.
To address these challenges, researchers propose a cost-aware single-generation architecture designed to meet the demands of enterprise-level nested schemas. This architecture includes schema selection, metadata retrieval, and error repair components, leveraging LLM-driven iterative schema discovery and structured error models to optimize SQL generation.
On the DevRev benchmark, the system achieved 91.7% answer correctness, 54.6 percentage points above the next-best baseline; on the Spider 2.0 Snowflake public dataset, the system is competitive at a single-generation operating point. The study demonstrates significant advantages of this new architecture in handling complex enterprise data, providing new directions for future research and applications.
Deep Analysis
Background
Natural language to SQL conversion technology has made significant progress in recent years, especially on academic benchmarks like Spider 1.0 and 2.0. However, real-world enterprise data often features graph-like, semi-structured, and deeply nested characteristics that are not adequately measured by existing benchmarks.
Core Problem
Existing NL2SQL systems face challenges when dealing with enterprise-level nested schemas. These schemas often include multi-layer nesting and polymorphic link relationships, requiring complex reasoning and SQL generation capabilities.
Innovation
The core innovation of this study is the introduction of a new architecture capable of handling the complexity of enterprise-level nested schemas. This architecture significantly improves SQL generation accuracy and efficiency through LLM-driven iterative schema discovery and structured error models.
Methodology
- �� Utilize LLM for iterative schema discovery to select appropriate database tables.
- �� Retrieve compressed schema for each table through tool calls.
- �� Optimize SQL generation process using structured error classification and repair mechanisms.
- �� Accumulate transferable SQL rules through a dynamic learning framework.
Experiments
Experiments used the DevRev NL2SQL benchmark and the Spider 2.0 Snowflake public dataset. The system achieved 91.7% answer correctness on 900 queries and was competitive at a single-generation operating point.
Results
The system achieved 91.7% answer correctness on the DevRev benchmark, 54.6 percentage points above the next-best baseline. It performed well on the Spider 2.0 Snowflake dataset at a single-generation operating point.
Applications
The system can be directly applied to scenarios requiring complex enterprise data processing, such as financial analysis and customer relationship management, significantly improving data query efficiency and accuracy.
Limitations & Outlook
While the system performs well on nested schemas, it may encounter performance bottlenecks with extremely complex nested structures. Additionally, computational costs need further optimization.
Plain Language Accessible to non-experts
Imagine a complex enterprise database as a multi-layered library, where each book has its own chapters and sub-chapters. Traditional SQL queries are like searching for a specific paragraph in a book, while our system acts like a smart librarian, quickly finding the needed information even if it's hidden across multiple layers of shelves. This way, the system efficiently handles complex query requests.
ELI14 Explained like you're 14
Imagine you're in a huge maze looking for something. Regular SQL queries are like having a simple map, which might not find all the secret passages. Our system is like a super-smart navigation assistant, helping you find all the hidden paths and quickly reach your destination! It's like having an unbeatable game guide that helps you breeze through levels!
Glossary
NL2SQL (Natural Language to SQL)
A technology that converts natural language into SQL queries, aimed at simplifying database querying.
Used to generate SQL queries to extract information from complex enterprise databases.
SDS (Semantic Depth Score)
A scoring standard used to assess the analytical reasoning depth of SQL queries.
Used to measure the complexity and accuracy of generated SQL queries.
ARRAY[STRUCT] (Array Structure)
A type of database field that allows for the storage of multi-level nested data.
Common in database schemas when handling enterprise-level nested schemas.
LATERAL FLATTEN
A SQL operation used to flatten nested array structures.
Used to handle complex nested data structures.
Polymorphic Link
A type of database relationship that allows linking to multiple entity types.
Common in handling complex enterprise-level data schemas to link different entities.
Open Questions Unanswered questions from this research
- 1 How to further optimize the system to handle more complex nested schemas.
- 2 Finding the optimal balance between computational cost and performance.
Applications
Immediate Applications
Enterprise Data Analysis
The system can be used for enterprise data analysis, improving query efficiency and accuracy, especially when handling complex data structures.
Long-term Vision
Intelligent Database Management
The system has the potential to become an intelligent database management tool, automating complex query processing and improving enterprise data management efficiency.
Abstract
Natural-language-to-SQL systems have ad- vanced rapidly on academic benchmarks, yet production enterprise schemas exhibit graph- like, semi-structured, deeply nested structure that current benchmarks do not measure. We make two complementary contributions. First, we introduce the DevRev NL2SQL bench- mark: 900 execution-verified queries with nested-type and link-graph structure, accom- panied by the Semantic Depth Score (SDS), a schema-agnostic rubric for analytical reasoning depth. Second, we present a cost-aware single- generation agentic architecture whose schema- selection, metadata-retrieval, and error-repair components are designed for the requirements this regime imposes. On the DevRev NL2SQL benchmark the system attains 91.7% answer correctness, a margin of 54.6 percentage points over the next-best baseline; on the Spider 2.0 Snowflake public dataset, it is competitive with leading systems at a single-generation operating point.