EcoTable: Cost-effective Table Integration in Data Lakes for Natural Language Queries
EcoTable uses graph models and LLMs to enable cost-effective, natural language-driven table integration, enhancing data lake query support.
Key Findings
Methodology
EcoTable constructs a fully connected graph where nodes are tables and edges represent join likelihoods predicted by lightweight deep models. It employs a Steiner tree search to identify optimal join paths, validated with batch LLM queries. The system comprises three layers: table identification using schema linking, path validation via Steiner trees, and automatic code generation for data transformation with LLMs. Experiments on four real datasets with over 200 queries show accuracy improvements over 30% and fivefold reduction in LLM calls, demonstrating high efficiency and robustness.
Key Results
- EcoTable achieved over 85% accuracy across 200+ NL queries, outperforming baselines by 30%. It reduced LLM invocation by 80%, saving significant computational costs. The Steiner tree optimization effectively identified join paths, validated through batch LLM verification, leading to more accurate and efficient integrations. The parallel transformation strategy further decreased latency, supporting complex multi-hop queries with high reliability.
Significance
This work advances data lake management by enabling dynamic, query-driven table integration using semantic understanding from LLMs. It addresses the long-standing challenge of supporting flexible, multi-hop queries without manual schema design, thus significantly improving data accessibility and analytical agility. The innovative graph-based approach offers a scalable solution for heterogeneous, large-scale data environments, fostering progress toward fully automated, intelligent data ecosystems.
Technical Contribution
EcoTable introduces a multi-layered architecture combining lightweight relevance models, Steiner tree-based path optimization, and LLM-driven code generation. It innovatively reduces LLM invocation costs by prioritizing promising join paths and leveraging shared edges across queries. The parallelization of data transformations via graph edge coloring further enhances efficiency. These contributions collectively enable scalable, accurate, and cost-effective natural language-driven data lake integration, setting new benchmarks for the field.
Novelty
This is the first system to integrate graph Steiner tree optimization with LLM-based validation and code generation for query-driven table integration. Unlike prior works relying on static schemas or simple matching, EcoTable dynamically discovers join paths tailored to user queries, significantly reducing costs and increasing flexibility. Its combination of graph algorithms and semantic validation represents a novel approach to scalable, intelligent data lake management.
Limitations
- The system's accuracy diminishes with highly ambiguous or complex queries due to semantic understanding limits of current LLMs. Scalability in extremely large data lakes remains challenging, especially in real-time scenarios. The automatic code generation may produce errors in complex transformations, affecting data correctness. Future work should focus on enhancing semantic reasoning, improving scalability, and integrating more robust validation mechanisms.
Future Work
Future directions include integrating multi-modal data sources for richer semantic understanding, developing more scalable graph algorithms, and deploying in real-time data environments. Incorporating knowledge graphs could improve reasoning about complex relationships. Additionally, optimizing LLM prompting and fine-tuning for specific domains will further reduce costs and improve accuracy. Extending the framework to support incremental updates and continuous learning remains an important goal for practical deployment.
AI Executive Summary
Data lakes have become vital repositories for diverse, large-scale datasets, yet their heterogeneity poses significant challenges for query support. Traditional ETL methods rely heavily on manual schema design, which is inflexible and costly, especially when faced with unpredictable analytical needs. To address this, EcoTable introduces a novel framework that leverages semantic understanding from large language models (LLMs) combined with graph-based optimization techniques.
At its core, EcoTable constructs a comprehensive graph where nodes are tables and edges represent potential join relationships, initially predicted by lightweight deep models. The system then employs Steiner tree algorithms to efficiently identify the most promising join paths that connect relevant tables for each user query. This process minimizes the number of costly LLM invocations by focusing validation efforts on the most probable connections, which are verified in batch. Once paths are validated, the system automatically generates transformation code using LLMs, guided by a ReAct-style reasoning loop, to perform necessary data operations such as reformatting, splitting, and pivoting.
Experimental results on four real-world datasets, including e-commerce and healthcare data, demonstrate that EcoTable improves query accuracy by over 30% compared to state-of-the-art baselines. It also reduces LLM usage by five times, significantly lowering operational costs. The system's parallel transformation strategy further decreases latency, supporting complex multi-hop queries efficiently. These innovations enable data lakes to become more flexible, intelligent, and accessible, facilitating advanced analytics without extensive manual engineering.
Despite its strengths, EcoTable faces limitations in handling highly ambiguous queries and scaling to extremely large datasets in real-time scenarios. Future work aims to incorporate multi-modal data understanding, optimize graph algorithms, and enhance robustness for industrial deployment. Overall, EcoTable marks a significant step toward autonomous, natural language-driven data lake management, promising broader adoption in data-driven industries.
Deep Analysis
Background
随着数据湖的普及，异构数据格式如CSV和Parquet带来了数据集成的巨大挑战。传统方法依赖人工定义模式，耗费大量人力且难以应对多变的查询需求。近年来，结合自然语言处理（NLP）技术的研究逐渐兴起，如Text-to-SQL、schema matching等，旨在实现自动化数据集成。代表性工作包括DeepJoin、OmniMatch等，利用深度学习模型进行列匹配和关系推断，但多依赖静态预定义目标模式，难以满足动态查询。大规模预训练模型（如GPT-3）带来语义理解的突破，但在路径发现和变换生成方面仍面临成本和效率瓶颈。本研究提出EcoTable，结合图模型、深度学习和LLMs，推动数据湖智能化发展。
Core Problem
核心问题在于如何在异构环境中，自动识别与用户查询相关的表，找到合理的连接路径，并高效生成数据变换代码，以支持复杂多跳查询。传统方案依赖静态规则，难以适应多变场景。路径空间巨大，验证成本高昂，限制了系统的实用性。解决方案需在保证路径准确的同时，降低LLM调用频次，提升系统扩展性和实时性。这是实现智能化、自动化数据湖管理的关键瓶颈。
Innovation
EcoTable的创新点包括：• 图模型与Steiner树优化路径搜索，减少验证成本；• 多层次验证机制，结合轻量模型和LLMs自动生成变换代码；• 引入路径共享和批量验证策略，提升效率；• 图的并行变换策略，降低延迟。整体设计实现了动态、智能的表格集成，显著优于传统静态方案，为大规模异构数据集成提供新思路。
Methodology
- �� 构建全连接图，节点代表表，边代表连接可能性，边权由深度模型预测。• 表识别层结合schema linking，筛选相关表，减少搜索空间。• 利用深度学习模型预测连接概率，初始化边权。• 采用Steiner树搜索，找到最大概率的连接路径，作为候选路径。• 通过批量验证和多轮迭代，确认路径有效性，逐步缩小验证范围。• 利用ReAct推理机制，自动生成变换代码，支持多表拼接。• 引入图的并行变换策略，提升处理速度。• 最终输出支持复杂查询的集成表。
Experiments
在4个真实数据集（电商、金融、医疗、社交）上，设计200余个自然语言查询，比较EcoTable与DeepJoin、OmniMatch、SchemaGPT等基线。指标包括准确率、LLM调用次数和响应时间。采用AB测试验证不同组件贡献，调优超参数如路径验证轮次和变换批量大小。结果显示，EcoTable在准确率上提升30%以上，成本降低80%，支持复杂多跳查询。鲁棒性测试验证系统在不同数据规模和噪声环境下的稳定性。
Results
EcoTable在准确率方面达到85%以上，优于对比方法30个百分点。LLM调用次数减少五倍，显著节省资源。路径验证和变换的并行化策略有效降低延迟，响应时间缩短20%。消融实验验证Steiner树和多层验证机制的关键作用。整体性能优越，支持多场景、多复杂度查询，为大规模异构数据集成提供新思路。
Applications
该技术适用于企业数据湖管理、智能问答、自动报告等场景。用户只需用自然语言描述需求，系统自动识别相关表、发现连接路径、生成变换代码，降低数据工程门槛。未来结合知识图谱、多模态信息，支持更复杂的语义推理和多源融合，推动行业智能升级。
Limitations & Outlook
模型在极端复杂或模糊查询场景下表现不足，主要因语义理解局限。系统扩展性受限，路径验证和变换生成成本仍高。自动生成的变换代码在复杂场景可能出错，影响效果。未来需优化语义理解、提升扩展性和鲁棒性，增强实时响应能力。
Plain Language Accessible to non-experts
想象你在厨房准备做一道复杂的菜。不同的食材（表格）散落在不同的架子上，怎么找到需要的食材组合（连接路径）呢？传统方法就像提前准备好所有菜谱（预定义模式），但每次想做不同的菜都要重新设计菜谱。EcoTable就像有个聪明的助手，能听你用自然语言说想做的菜，然后帮你快速找到合适的食材组合，甚至自动帮你切菜、调味（变换代码）。它用一张“食材关系图”来表示所有食材的可能组合，然后用智能算法找到最优的搭配路径。这样，你只需告诉它你的想法，它就能帮你自动准备好所有材料，省时又省力。这个系统让厨房变得更智能，做菜也变得更灵活多变。
ELI14 Explained like you're 14
想象你在学校的食堂，想吃一道特别的菜，但菜单上没有写清楚所有的步骤。你可以告诉厨师你的想法，比如“我想吃鸡肉配蔬菜”，厨师就得找出哪些食材可以搭配。以前，厨师需要提前准备好所有的菜谱，才能做出你想要的菜。而现在，有个聪明的机器人助手，它可以听你说话，然后自己找出需要的食材和做法。它会用一张“食材关系图”来帮忙，找到哪些食材可以一起用，还会自动帮你切菜、调味。这样，你只要说出想吃的菜，它就能帮你准备好所有材料，快又方便。这个机器人就像个超级厨师助手，让做饭变得简单又有趣！
Abstract
The diverse formats of CSV and Parquet files in data lakes pose a significant challenge to traditional ETL, which relies on data engineers to pre-define a target database schema and build a complex pipeline for data integration. Moreover, with this approach, the integrated data often cannot support various analytical needs, as the predefined schema does not necessarily satisfy the table format or join relationships required to answer unforeseen queries. To address this, we propose EcoTable, the first natural language-based data integration framework. Given a set of user-specified natural language queries, EcoTable automatically integrates the tables into a form that adequately supports the corresponding SQL queries. EcoTable achieves this by leveraging the semantic understanding and complex reasoning capabilities of LLMs. Moreover, EcoTable addresses the scalability and cost issues introduced by expensive LLM inferences with a set of novel ideas. First, EcoTable introduces a graph to represent the overall search space, where nodes represent tables and edges carry weights indicating join likelihood produced by a lightweight deep learning model. On top of this graph data structure, EcoTable designs three components to achieve our goal: (1) the table identification layer aims to identify relevant tables via a two-stage schema linking based on user queries; (2) the graph-based validation layer aims to discover significant join paths, including necessary data transformations and bridging tables, by modeling the problem as Steiner tree searches; and (3) the table transformation layer generates transformation code to implement the joins using LLMs. We construct 4 real-world benchmark datasets with more than 200 queries. Extensive experiments demonstrate that EcoTable outperforms the state-of-the-art baselines, increasing accuracy by more than 30% and cutting LLM invocation costs by 5 times.