Both Ends Count! Just How Good are LLM Agents at "Text-to-Big SQL"?
A ReAct agent with VES* and VCES shows that GPT-4o can trade about 7% accuracy for up to 12.16× speed.
Key Findings
Methodology
The paper evaluates Text-to-Big SQL as a systems problem, not merely a translation task. A zero-shot ReAct agent uses an LLM controller and four Spark SQL tools: list_tables, get_schema, check_query, and run_query. Beyond Execution Accuracy, the framework models column-level precision, end-to-end latency, and execution cost through VES*, VCES, and CVQ, thereby measuring correctness, interaction overhead, scalability, and cloud economics.
Key Results
- On selected high-accuracy BIRD queries, GPT-4o achieved EX 0.93 with 6.55 seconds mean end-to-end time. Gemini 3 Pro reached EX 1.00 but required 54.55 seconds, while GLM-5 also reached 1.00 but required 79.63 seconds. Accuracy alone therefore obscures major deployment differences.
- Normalized VES* ranked GPT-4o at 1.00, Gemini 3 Flash at 0.81, Claude Opus 4.6 at 0.51, and Gemini 3 Pro at 0.23. Its dispersion was 809.09%, compared with only 54.93% for VES, demonstrating stronger discrimination among accurate agents.
- At scale, GPT-4o reportedly compensates for roughly 7% lower accuracy than later-generation leaders with up to a 12.16× speedup. GPT-5.2 is more than twice as cost-effective as Gemini 3 Pro at large input scales, emphasizing that model choice depends on workload economics.
Significance
The study reframes Text-to-SQL as an end-to-end data-system problem. In Big Data, a wrong join or unnecessary projection can trigger massive scans, long waits, and cloud charges even when the final answer appears nearly correct. Academically, the work connects NLP evaluation with database execution and cost modeling. Operationally, it provides a principled basis for choosing models, routing agent stages, controlling retries, and designing interactive analytics systems.
Technical Contribution
The principal contribution is a metric family that combines partial correctness with runtime and cost. Column precision P(S,Ŝ)=|S∩Ŝ|/|Ŝ| penalizes irrelevant projected fields without discarding salvageable outputs. VES* multiplies validity, column precision, and Tgold/Te2e; VCES additionally divides by Ce2e; CVQ estimates retry-until-success cost as Ce2e/p. Stage-level traces further expose list, schema, check, and run bottlenecks.
Novelty
Rather than proposing another SQL generator, the paper systematically treats query generation and query execution as co-equal evaluation targets. Relative to EM, EA, and conventional VES, it integrates agent actions, reasoning latency, projection overhead, scalable data volume, and monetary cost. This produces a production-oriented definition of Text-to-Big SQL and reveals trade-offs that standard translation benchmarks cannot see.
Limitations
- The evaluation centers on Spark SQL, an AWS m5.xlarge instance, and a zero-shot ReAct architecture. It does not systematically cover Athena, BigQuery, distributed-cluster effects, caching, network variability, concurrency, or file-format-specific behavior.
- Only BIRD and scalable TPC-H are used. Results depend on prompts, provider APIs, pricing, and low-latency reasoning settings, so model rankings may not transfer directly to every enterprise schema or mixed workload.
- VES* uses a fixed multiplicative structure and implicit weighting among validity, column precision, and time. The paper does not establish that this formula always matches user utility, service-level objectives, or business-specific cost functions.
Future Work
Future research should assign different models to controller, checker, and individual ReAct stages, and compare model ensembles, caching, query-plan optimization, and adaptive stopping. Broader studies should use Athena, BigQuery, distributed Spark, streaming data, and multi-tenant workloads. Learned metrics could incorporate workload-specific budgets, latency targets, risk tolerance, and human satisfaction rather than fixed weights.
AI Executive Summary
Natural-language database access has advanced rapidly, but most Text-to-SQL benchmarks still ask a narrow question: did the generated query return the expected answer? This paper argues that the question becomes inadequate when queries operate over Big Data. A redundant column, inefficient join, or failed retry may scan vast files, delay an interactive session, and increase a cloud bill. The authors call this broader setting Text-to-Big SQL and insist that both ends of the pipeline count.
Their test system is a zero-shot ReAct agent. An LLM controller invokes list_tables, get_schema, check_query, and run_query around a Spark SQL backend, while the evaluation records reasoning, tool use, execution, and cost. VES* extends efficiency evaluation with column-level precision and total end-to-end time; VCES adds monetary cost; CVQ estimates the expected expense of retrying until a valid answer appears. These metrics distinguish harmless extra output from damaging row-count or join errors.
The results are striking. On selected BIRD queries, GPT-4o achieved EX 0.93 in 6.55 seconds, whereas Gemini 3 Pro achieved EX 1.00 in 54.55 seconds and GLM-5 required 79.63 seconds. Normalized VES* was 1.00 for GPT-4o but only 0.23 for Gemini 3 Pro; its dispersion reached 809.09%, compared with 54.93% for VES. The study reports that GPT-4o trades roughly 7% accuracy for up to 12.16× speed, while GPT-5.2 is over twice as cost-effective as Gemini 3 Pro at large scales. The broader message is practical: a useful database agent must be correct, fast, economical, and conservative about data movement.
Deep Analysis
Background
Text-to-SQL has traditionally relied on Exact Match, Execution Accuracy, and Valid Efficiency Score. Production LLMs such as GPT, Gemini, and Claude, combined with ReAct scaffolding, can inspect schemas and generalize across databases. Yet Big Data engines such as Spark SQL, Athena, and BigQuery introduce scan volume, latency, and billing effects that grow with data size. Existing benchmarks largely ignore these system-level consequences.
Core Problem
The problem is to evaluate an agent that translates natural language and then executes the result at scale. Wrong row counts or joins invalidate outputs; missing columns require re-execution; extra columns may be manually removed but still increase scan and transfer cost. Moreover, LLM reasoning and query checking can be slower than the database itself. A binary accuracy score cannot represent these distinct failure modes.
Innovation
- �� Treat generation and execution as equal evaluation endpoints.
- �� Use column precision to preserve partial correctness while penalizing redundant projections.
- �� Measure Te2e across LLM, tools, and Spark execution.
- �� Introduce VES*, VCES, and CVQ for efficiency, cost, and retry economics.
- �� Aggregate traces by list, schema, check, and run stages to expose actionable bottlenecks.
Methodology
- �� Input: natural-language BIRD questions and scalable TPC-H workloads.
- �� Controller: a production LLM follows ReAct Thought, Action, and Observation cycles.
- �� Tools: list_tables executes SHOW TABLES; get_schema uses SHOW CREATE TABLE and optional sampling; check_query applies LLM-assisted syntax checks; run_query executes SQL in Spark.
- �� Termination: the agent stops after the first run_query to avoid costly execution loops.
- �� Metrics: an output-validity indicator is multiplied by P(S,Ŝ) and Tgold/Te2e; VCES additionally divides by Ce2e, while CVQ estimates Ce2e/p under geometric retries.
Experiments
Experiments ran on an AWS us-east-1 m5.xlarge using official model APIs and provider token prices. The benchmarks were BIRD and TPC-H. Evaluated systems included GPT-4o, GPT-5, GPT-5.2, Gemini 2.5 Flash, Gemini 3 Flash, Gemini 3 Pro, Claude Opus 4.5 and 4.6, Kimi K2.5, and GLM-5. Eight BIRD queries with at least 0.85 average EX for most models were selected to isolate efficiency differences.
Results
GPT-4o obtained EX 0.93 and 6.55-second mean Te2e; Gemini 3 Flash obtained 1.00 and 8.37 seconds; Opus 4.6 obtained 1.00 and 12.60 seconds; Gemini 3 Pro obtained 1.00 and 54.55 seconds; GLM-5 obtained 1.00 and 79.63 seconds. The check stage generally dominated runtime. VES* separated systems much more sharply than VES, with 809.09% versus 54.93% dispersion, showing that interaction and cost matter after accuracy saturates.
Applications
The framework applies to enterprise natural-language analytics, lakehouse querying, interactive BI, and cloud SQL gateways. Operators can choose models using latency and budget targets, route different stages to different models, and use CVQ to forecast retry costs. Stage traces can also identify whether schema access, reasoning, validation, or physical execution is the dominant bottleneck.
Limitations & Outlook
The study uses one primary backend, one machine configuration, and zero-shot prompts; clusters, caching, concurrency, networks, and file formats may change outcomes. BIRD is relatively small, while TPC-H does not capture every enterprise workload. The fixed multiplicative form of VES* may not fit all business utilities. Future work should evaluate multiple engines and clouds, learn workload-specific weights, and study dynamic routing and budget-aware stopping.
Plain Language Accessible to non-experts
Imagine a restaurant receiving orders from thousands of customers. A diner says, “Bring me the blue-eyed superheroes,” and the waiter must first inspect the menu, locate the right ingredients, write an order, check it, and send it to the kitchen. Older evaluations only ask whether the final dish is correct. But in a huge restaurant, we also care whether the waiter took too long, asked unnecessary questions, made the kitchen prepare unwanted side dishes, or caused an expensive waste of ingredients.
VES* is like a combined restaurant score. It checks whether the main dish is right, how much irrelevant garnish was prepared, and how long the entire process took. VCES also counts the bill, while CVQ estimates what repeated re-orders will cost when the first attempt fails.
In the experiments, GPT-4o behaved like a quick waiter: EX was 0.93 and average completion took 6.55 seconds. Gemini 3 Pro reached perfect EX, but took 54.55 seconds; GLM-5 took 79.63 seconds. The lesson is simple: in a giant restaurant, perfect food is not enough if every customer waits forever. A good AI must be accurate, quick, economical, and careful not to prepare what nobody ordered.
ELI14 Explained like you're 14
Think of an AI agent as a game teammate searching a giant warehouse. You say, “Find how many superheroes have blue eyes.” The teammate cannot just guess. It checks which boxes exist, reads the labels, writes a search command, asks another helper to check the command, and finally makes the warehouse machines search.
Old tests mainly ask, “Did the teammate get the right number?” But what if it searched every box in the warehouse, waited forever, or spent all your game coins? That answer might be correct, yet still be a terrible teammate. This paper uses ReAct, where the agent keeps cycling through thinking, acting, and observing, while measuring the time and resources used at every step.
The results are surprising! GPT-4o finished the selected BIRD tasks in 6.55 seconds with EX 0.93. Gemini 3 Pro got EX 1.00, but took 54.55 seconds; GLM-5 also got 1.00, but took 79.63 seconds. So the perfect-score player is not automatically the best choice for a live game.
The big idea is that future AI should not only say, “I got the answer!” It should also say, “I got it quickly, avoided useless searching, and did not waste your budget.” That matters even more when the warehouse becomes enormous.
Glossary
Text-to-Big SQL
Natural-language-to-SQL systems evaluated under large-scale execution constraints. It extends Text-to-SQL by including latency, resource use, scalability, and monetary cost.
The paper’s central problem formulation.
ReAct
A reasoning-and-acting framework that alternates Thought, Action, and Observation. It lets an LLM select tools, interpret feedback, and refine its next step.
The zero-shot agent architecture.
VES*
A proposed efficiency score combining output validity, column precision, and the ratio Tgold/Te2e. It gives partial credit for usable results while penalizing slow interaction and extra columns.
The principal Text-to-Big SQL metric.
VCES
A cost-oriented extension of VES* that also divides by total end-to-end cost Ce2e. It is designed for cloud deployments with usage-based billing.
Used to compare economic efficiency.
CVQ
Expected Cost per Valid Query, defined as Ce2e/p, where p is single-shot validity. It estimates average spending under retry-until-success behavior.
Measures the cost of failed attempts.
Column precision
P(S,Ŝ)=|S∩Ŝ|/|Ŝ|, the fraction of returned columns that are relevant. It penalizes superfluous projections without treating them as fully invalid.
A component of VES* and VCES.
Open Questions Unanswered questions from this research
- 1 How should metric weights change across workloads? Interactive analytics may prioritize latency, whereas batch processing may prioritize scan cost; the fixed multiplicative form has not been validated as a universal utility model.
- 2 Do model rankings remain stable under caching, concurrency, network variance, distributed clusters, and multi-tenant interference? Cross-cloud and production traces are needed to answer this.
Applications
Immediate Applications
Enterprise natural-language analytics
Data teams can connect a ReAct agent to Spark SQL, use VES* to compare accuracy and response time, and use VCES or CVQ to estimate bills. This supports evidence-based selection between fast models such as GPT-4o and more accurate but slower alternatives.
Cloud SQL gateway
A gateway can log schema, checking, and execution stages, enforce scan or cost thresholds, and request rewrites when generated SQL projects too many fields. This reduces failed queries, unnecessary data movement, and uncontrolled billing.
Long-term Vision
Adaptive model routing
Future systems could assign specialized models to schema discovery, syntax checking, and complex reasoning, switching policies according to data volume, budget, latency objectives, and risk. Such routing may outperform a single LLM across heterogeneous workloads.
Abstract
Text-to-SQL and Big Data are both extensively benchmarked fields, yet there is limited research that evaluates them jointly. In the real world, Text-to-SQL systems are often embedded with Big Data workflows, such as large-scale data processing or interactive data analytics. We refer to this as ``Text-to-Big SQL''. However, existing text-to-SQL benchmarks remain narrowly scoped and overlook the cost and performance implications that arise at scale. For instance, translation errors that are minor on small datasets lead to substantial cost and latency overheads as data scales, a relevant issue completely ignored by text-to-SQL metrics. In this paper, we overcome this overlooked challenge by introducing novel and representative metrics for evaluating Text-to-Big SQL. Our study focuses on production-level LLM agents, a database-agnostic system adaptable to diverse user needs. Via an extensive evaluation of frontier models, we show that text-to-SQL metrics are insufficient for Big Data. In contrast, our proposed text-to-Big SQL metrics accurately reflect execution efficiency, cost, and the impact of data scale. For example, GPT-4o compensates for roughly 7% lower accuracy than the top-performing later-generation models with up to a 12.16x speedup, while GPT-5.2 is more than twice as cost-effective as Gemini 3 Pro at large input scales.