TAPEX: Table Pre-training via Learning a Neural SQL Executor
TAPEX learns a neural SQL executor from synthetic SQL-output pairs and sets SOTA on WikiSQL, WTQ, SQA, and TabFact.
Key Findings
Methodology
TAPEX reframes table pre-training as execution learning. Starting from about 1,500 semi-structured tables sampled from the WIKITABLEQUESTIONS training split, it automatically instantiates SQL templates extracted from SQUALL, such as SELECT num1 WHERE text1=val1, by filling column headers and cell values. This yields up to 5 million executable SQL-query/answer pairs. A BARTLarge encoder-decoder then consumes the linearized query plus table and is trained to generate the gold execution result, effectively learning a neural SQL executor rather than a masked-language reconstructor.
Key Results
- On WIKISQL-WEAK, TAPEX reaches 89.5% test denotation accuracy, outperforming BART by 3.7 points and the previous best by 2.3 points, despite the latter already using execution-guided decoding. The gain shows that execution-oriented pretraining transfers beyond inference-time heuristics.
- On WIKITABLEQUESTIONS, TAPEX achieves 57.5% test denotation accuracy, a +4.8 point improvement over the previous best and +19.5 over vanilla BART (38.0%). This is especially striking in a low-data, compositional-reasoning setting where table structure matters more than surface form.
- On SQA, TAPEX obtains 48.4% conversation-level and 74.5% sentence-level accuracy; on TabFact, it reaches 84.2% overall and 79.6% on Testcomplex. The model also correctly executes 89.6% of nearly 20,000 held-out SQL queries, with strong performance on Filter, Aggregate, Superlative, Arithmetic, and Comparative operators.
Significance
The paper shows that table pre-training need not rely on massive noisy web mining or labor-intensive template writing. Instead, high-quality synthetic executable programs can provide scalable supervision that teaches models to reason over structure, values, and operations. This is important academically because it introduces a clean execution-centric objective for structured data learning. It is equally important practically because it offers a controllable, data-efficient route toward table QA, fact verification, analytics assistants, and other table-centric NLP systems.
Technical Contribution
TAPEX contributes three concrete advances. First, it introduces execution-centric pre-training: the model is optimized to reproduce SQL execution outputs, aligning learning directly with structured reasoning. Second, it builds a large, diverse synthetic corpus via automatic SQL template instantiation and empty-result filtering, avoiding the noise of web-crawled corpora and the brittleness of hand-written templates. Third, it reformulates downstream TableQA and TableFV as sequence generation, which makes one encoder-decoder architecture usable across answer generation and entailment/refutation, and enables straightforward multi-task fine-tuning.
Novelty
This is, to the authors’ knowledge, the first work to exploit synthetic executable programs for table pre-training and achieve new SOTA on multiple benchmarks. Unlike TAPAS-style whole-word masking or TABERT-style masked column prediction, TAPEX does not treat tables as just another textual format. It directly supervises the model with executable semantics, approximating the behavior of a SQL engine.
Limitations
- The synthetic corpus is template-driven and mostly based on semi-structured public tables, so coverage of nested queries, cross-table joins, and open-domain knowledge reasoning remains limited. The learned executor is strong on the sampled operator space but not proven to generalize broadly beyond it.
- The approach still depends on a large BART model and nontrivial compute: pre-training runs up to 50k steps with batch size 256 and takes about 36 hours on 8 Tesla V100 GPUs. Robustness to messy real-world tables and formatting variation is also not fully resolved.
Future Work
Likely extensions include richer SQL/program synthesis, more diverse table sources, and stronger evaluation on complex compositional queries. A natural next step is to scale the idea to larger seq2seq backbones and combine it with retrieval, tool use, and agentic table operations to build more general data-analysis foundation models.
AI Executive Summary
TAPEX redefines a stubborn problem in table understanding: rather than searching for ever-larger piles of table-text pairs, it asks whether a model can learn to behave like a SQL engine. If a model can take a table and a query, then reliably produce the same output as an actual executor, it should possess a deep grasp of structure, cell values, and operations.
The method is elegant. Using BARTLarge as the backbone, TAPEX synthesizes a large execution corpus by sampling about 1,500 training tables from WIKITABLEQUESTIONS and instantiating SQL templates extracted from SQUALL. Templates such as SELECT num1 WHERE text1=val1 are filled with sampled headers and cell values to form concrete queries. The table is linearized with special markers like [HEAD] and [ROW], appended to the query, and the model is trained to generate the executor-produced answer. Downstream TableQA and TableFV are also cast into a generative formulation.
The results are decisive. TAPEX achieves 89.5% on WIKISQL-WEAK, 57.5% on WIKITABLEQUESTIONS, 74.5% sentence-level and 48.4% conversation-level accuracy on SQA, and 84.2% on TabFact, all new bests at the time. Against vanilla BART, the gain on WIKITABLEQUESTIONS is a dramatic 19.5 points, underscoring how much execution supervision helps in low-resource compositional reasoning. The model also correctly executes 89.6% of nearly 20,000 held-out SQL queries, suggesting that it has internalized meaningful table operations rather than merely memorizing surface patterns.
Beyond the headline numbers, TAPEX offers a broader message: structured data can be taught through synthetic programs, not only noisy text. That opens a practical route to stronger table QA systems, fact checkers, and business intelligence copilots. Perhaps most importantly, it suggests that executable semantics may be a more efficient learning signal than reconstruction for many structured domains.
The main caveat is scope. The corpus is template-based, the tables are semi-structured, and the evaluated tasks, while diverse, still live within a constrained operator space. Extending this paradigm to joins, nested logic, messy enterprise tables, and broader tool use will be the real test. Still, TAPEX provides a compelling blueprint for teaching language models to reason over tables by learning to execute them.
Deep Analysis
Background
Table understanding sits at the intersection of NLP, databases, and semantic parsing. After BERT and BART showed that large-scale pre-training can transform free-form language tasks, researchers began adapting pre-training to tables. TAPAS used whole-word masking; TABERT proposed Masked Column Prediction; other approaches mined tables and nearby text from the web or generated synthetic sentences from tables. These methods were useful but imperfect: web data is noisy and expensive to clean, while template-based generation often lacks diversity and requires heavy manual effort. TAPEX enters this landscape by exploiting a property unique to tables: executability.
Core Problem
The core problem is how to pre-train a model to truly understand tables when high-quality tabular corpora are scarce. Tables are not just text arranged in rows and columns; they encode structure, alignment, aggregation opportunities, and numeric relations. Existing reconstruction objectives are often too weak to teach these skills efficiently, so they need huge corpora and still may not transfer well. TAPEX targets the missing supervision signal: execution behavior.
Innovation
TAPEX’s first innovation is the execution-centric objective. Instead of masking spans or columns, it trains the model to emit the result of a SQL query, directly aligning learning with table semantics.
Second, it uses synthetic executable programs. SQL templates are automatically extracted from SQUALL, instantiated on semi-structured tables, and filtered to remove empty-result queries. This creates a diverse corpus with controllable quality and scale.
Third, it unifies downstream tasks under generation. TableQA becomes answer generation; TableFV becomes sequence-based classification. This design makes multi-task learning natural and allows the same architecture to transfer across very different outputs.
Fourth, it demonstrates that a general-purpose encoder-decoder, specifically BARTLarge, can absorb table-specific reasoning without bespoke masking machinery, making the method simple to implement and broadly adaptable.
Methodology
- �� Backbone: BARTLarge, a standard Transformer encoder-decoder with 12 layers on each side, is used as the pre-training and fine-tuning model.
- �� Input formatting: the query text and a linearized table are concatenated. Tables are flattened with explicit boundary tokens such as [HEAD] and [ROW], and column/cell separators preserve row-column structure.
- �� Corpus synthesis: approximately 1,500 tables are sampled from WIKITABLEQUESTIONS training data; SQL templates are extracted from SQUALL and then instantiated by sampling headers and cell values from those tables.
- �� Supervision signal: an off-the-shelf SQL engine computes the gold execution result, which becomes the decoder target. In effect, the model learns to imitate a database executor.
- �� Filtering: queries with empty outputs are discarded because they provide little supervision about table executability.
- �� Fine-tuning: TableQA outputs the answer string autoregressively; TableFV uses the same input but predicts entailment/refutation with a binary classifier over the decoder’s final hidden state.
- �� Multi-task option: because all tasks share the same seq2seq interface, TAPEX can be further adapted via vanilla or multi-task fine-tuning, following the paper’s BART/T5-inspired setup.
Experiments
The paper evaluates on four benchmarks: WIKISQL-WEAK, WIKITABLEQUESTIONS, SQA, and TABFACT. For TableQA, the metric is denotation accuracy; for TabFact, it is classification accuracy. Pre-training uses up to 5 million SQL/result pairs, 50,000 steps, batch size 256, and a learning rate of 3e-5. Fine-tuning uses up to 20,000 steps, batch size 128, and the same learning rate. The implementation is built on fairseq, and the best checkpoint is selected by validation loss. Results are reported as medians over five random runs.
Results
The headline result is broad, consistent improvement. TAPEX reaches 89.5% on WIKISQL-WEAK, 57.5% on WIKITABLEQUESTIONS, 74.5% sentence-level and 48.4% conversation-level on SQA, and 84.2% on TabFact. Compared with BART, the gains are especially large on TableQA: +3.7 on WikiSQL-WEAK and +19.5 on WIKITABLEQUESTIONS. The paper also reports that TAPEX correctly executes 89.6% of held-out SQL queries, indicating genuine executor-like behavior. Operator-level analysis shows strong lifts on Select, Filter, Aggregate, Superlative, Arithmetic, Comparative, and Group questions.
Applications
In practice, TAPEX is relevant anywhere a user asks questions over tables and expects concise answers or truth judgments. That includes business intelligence dashboards, spreadsheet assistants, financial reporting tools, scientific database QA, and fact-checking systems. Because the architecture is generative and task-agnostic, it can be adapted to both answer extraction and binary verification without redesigning the model head. Organizations with structured data and moderate annotation budgets could use it as a pre-trained table reasoning layer.
Limitations & Outlook
TAPEX assumes a reasonably executable table environment and relies on SQL-template coverage, so truly open-ended reasoning, joins across multiple tables, and complex nested queries are not fully addressed. The synthetic corpus is also drawn from semi-structured public tables, leaving robustness on messy enterprise tables and unusual formats unresolved. While the compute cost is moderate by modern standards, it is still nontrivial. Future improvements likely require richer program synthesis, more diverse data, and integration with retrieval or tool-use pipelines.
Plain Language Accessible to non-experts
Imagine a big library where every shelf is a table. Some tables tell you who did what, some list prices, some record dates, and some compare facts. Now imagine you want a robot librarian that can answer questions like “Who was 24 years old besides Reyna Royo?” or “What year did Greece last host the Summer Olympics?” A naive robot might just stare at the shelves and guess. TAPEX teaches the robot a better trick: learn to follow a reliable recipe for finding the answer.
The recipe comes from SQL, a language computers use to ask databases for information. Instead of making people write thousands of training examples by hand, the researchers let the computer generate them automatically. They take a table, plug in a query pattern, and use a real database engine to produce the correct answer. Then the robot practices over and over: table in, query in, answer out.
Why does this help? Because a good robot librarian should not just recognize words; it should know how to search rows, compare numbers, add values, and find the biggest or smallest item. TAPEX gives it exactly that kind of practice. That is why, when tested on four different table tasks, it did better than older methods. In short, TAPEX teaches a machine not just to read the library, but to actually use it.
ELI14 Explained like you're 14
Picture your school’s gradebook as a giant table. One row is a student, one column is math score, another is age, another is hometown. Now imagine a teacher asks: “Who else is 24 years old?” or “What is the difference between Group A and Group B?” That’s not a simple memorization game — you have to look at the right row, the right column, and sometimes do a little math. Tough, right?
TAPEX is basically training an AI to become really good at this kind of table detective work. But instead of giving it a tiny stack of hand-written practice questions, the researchers built an automatic question factory. The factory makes lots of table questions using SQL, which is a computer language for asking data questions. Every question comes with the correct answer because a real database engine solves it first.
After tons of practice, the AI starts to get the hang of things. It learns how to find matching cells, compare values, pick the latest or biggest item, and even do arithmetic. That’s why it beats older models on WikiSQL, WikiTableQuestions, SQA, and TabFact. It’s not just guessing anymore — it’s learning the rules of the game.
So if you think of a table like a puzzle board, TAPEX is the AI that learned the puzzle strategy instead of memorizing one puzzle at a time. Pretty cool, right?
Glossary
SQL executor
A program that takes a SQL query and computes the answer from a database or table. In plain terms, it is a machine that “runs the query”; technically, it parses and executes structured operations such as selection, aggregation, and comparison.
TAPEX is trained to imitate SQL executor outputs during pre-training.
Denotation accuracy
A metric that checks whether the predicted answer is semantically equal to the gold answer, even if the surface form differs. This is useful when answers may be normalized or formatted differently.
Used for WikiSQL-WEAK, WikiTableQuestions, and SQA.
Sequence-to-sequence model
A model that maps one sequence into another sequence, typically with an encoder and an autoregressive decoder. In TAPEX, BARTLarge generates answer strings token by token.
The unified formulation for both pre-training and downstream tasks.
Whole-word masking
A pre-training objective that hides complete words and asks the model to reconstruct them. It helps language understanding but provides limited structural supervision for tables.
A key baseline idea in TAPAS-style table pre-training.
Masked Column Prediction
A table pre-training task that hides column names or types and asks the model to recover them. It emphasizes structure awareness but still remains a reconstruction-style objective.
The main pre-training idea in TABERT.
Open Questions Unanswered questions from this research
- 1 How far can execution-based pre-training generalize beyond template-generated SQL to genuinely open-ended, nested, or multi-table reasoning? The paper does not fully answer whether executor imitation scales to the full space of real-world analytical queries.
- 2 What is the right balance between synthetic program diversity and semantic fidelity? More varied templates may improve coverage, but it is unclear which kinds of generated programs best transfer to downstream tasks and which merely inflate training size.
Applications
Immediate Applications
Table QA systems
Teams building spreadsheet assistants, BI copilots, or data portal search tools can fine-tune TAPEX-style models to answer natural-language questions over tables. The key requirement is a structured table input and answerable queries.
Fact verification over tables
Newsrooms, compliance teams, and analytics workflows can use TAPEX as a binary verifier that checks whether a claim is supported by a table. It is especially useful when the claim can be grounded in explicit rows and columns.
Long-term Vision
Table foundation models
A longer-term vision is a general-purpose model that can query, explain, compare, and manipulate tabular data with tools. Realizing this will require richer synthetic programs, broader table sources, and tighter integration with retrieval and agentic tool use.
Abstract
Recent progress in language model pre-training has achieved a great success via leveraging large-scale unstructured textual data. However, it is still a challenge to apply pre-training on structured tabular data due to the absence of large-scale high-quality tabular data. In this paper, we propose TAPEX to show that table pre-training can be achieved by learning a neural SQL executor over a synthetic corpus, which is obtained by automatically synthesizing executable SQL queries and their execution outputs. TAPEX addresses the data scarcity challenge via guiding the language model to mimic a SQL executor on the diverse, large-scale and high-quality synthetic corpus. We evaluate TAPEX on four benchmark datasets. Experimental results demonstrate that TAPEX outperforms previous table pre-training approaches by a large margin and achieves new state-of-the-art results on all of them. This includes the improvements on the weakly-supervised WikiSQL denotation accuracy to 89.5% (+2.3%), the WikiTableQuestions denotation accuracy to 57.5% (+4.8%), the SQA denotation accuracy to 74.5% (+3.5%), and the TabFact accuracy to 84.2% (+3.2%). To our knowledge, this is the first work to exploit table pre-training via synthetic executable programs and to achieve new state-of-the-art results on various downstream tasks. Our code can be found at https://github.com/microsoft/Table-Pretraining.