Thucy: An LLM-based Multi-Agent System for Claim Verification across Relational Databases
Thucy employs a multi-agent architecture with LLMs to verify claims across multiple relational databases, achieving 94.3% accuracy on TabFact.
Key Findings
Methodology
Thucy integrates GPT-4-based multi-agent components—Data Expert, Schema Expert, and SQL Expert—using the MCP protocol for environment-agnostic database discovery and reasoning. The system autonomously explores unknown databases, extracts schemas, and generates SQL queries through multi-turn interactions, guided by toolsets. The Verifier orchestrates these agents, iteratively refining queries to produce a final verdict with supporting SQL evidence. This architecture ensures transparency and reproducibility, validated on TabFact with a 94.3% accuracy, surpassing previous models at 88.7%.
Key Results
- Achieved 94.3% accuracy on TabFact, outperforming prior models by 5.6 percentage points, demonstrating robust cross-database verification capabilities.
- Automatically discovers relevant tables and data, generating precise SQL queries for complex multi-table, multi-database claims.
- Provides detailed SQL queries as evidence, enhancing interpretability and trustworthiness of verification results.
Significance
This work advances automated fact verification by enabling fully autonomous, cross-database reasoning without prior environment knowledge. It addresses longstanding challenges in integrating heterogeneous structured data, offering significant impact for fact-checking, policy analysis, and data integrity assurance. The approach bridges the gap between large language models’ reasoning power and real-world data complexity, setting a new benchmark for structured data validation.
Technical Contribution
The paper introduces a novel multi-agent framework leveraging the MCP protocol for environment-agnostic database interaction, combined with multi-turn SQL optimization. It demonstrates how large models can coordinate specialized agents to explore, understand, and query unknown relational data sources, providing transparent, reproducible verification workflows. This design significantly enhances scalability, flexibility, and interpretability over prior static or single-database methods.
Novelty
First to realize a fully autonomous, cross-database fact verification system that combines multi-agent orchestration with large model reasoning. Unlike previous work limited to single tables or static environments, Thucy dynamically explores and verifies claims across multiple relational sources, delivering comprehensive, explainable results.
Limitations
- Relies on high-quality, well-structured databases; performance may degrade with noisy or incomplete data.
- Multi-database queries can be computationally intensive, affecting real-time applicability.
- Currently validated mainly on TabFact; real-world deployment requires adaptation to diverse, industry-specific data schemas.
Future Work
Future directions include supporting non-relational data sources, integrating reinforcement learning for query optimization, and developing user-interactive interfaces for validation refinement. Expanding to real-time data streams and heterogeneous data types will further enhance practical deployment, making automated fact verification more versatile and scalable.
AI Executive Summary
In an era where information overload makes truth verification increasingly challenging, existing solutions are often limited to small, static datasets. These systems struggle to handle the complexity and scale of real-world data environments, leaving many claims unverified. Addressing this gap, Thucy introduces a novel multi-agent framework powered by large language models, capable of autonomously exploring, understanding, and verifying claims across multiple relational databases.
The architecture comprises three specialized agents—Data Expert, Schema Expert, and SQL Expert—coordinated by a Verifier that manages multi-turn interactions. Using the Model Context Protocol (MCP), Thucy seamlessly connects to diverse databases without prior knowledge, dynamically discovering schemas, extracting relevant data, and generating SQL queries. This process is iterative, with each agent refining its output based on feedback, ensuring high accuracy and transparency.
Experimental results on the TabFact dataset demonstrate the system’s effectiveness, achieving a 94.3% accuracy rate—significantly surpassing previous models. The system not only verifies claims but also provides detailed SQL queries as evidence, enhancing interpretability and trust. Such capabilities are crucial for applications like public data validation, policy assessment, and fact-checking, where transparency and reproducibility are vital.
Thucy’s innovation lies in its environment-agnostic, fully autonomous exploration and reasoning, enabled by multi-agent coordination and advanced prompting strategies. This approach opens new avenues for automated, scalable, and explainable data verification, addressing longstanding challenges in structured data analysis. Looking ahead, future work aims to incorporate heterogeneous data sources, optimize query processes with reinforcement learning, and develop user-friendly interfaces, pushing the system closer to real-world deployment. Overall, Thucy marks a significant step forward in leveraging AI for trustworthy information management.
Deep Analysis
Background
The rapid growth of structured data from governmental, commercial, and scientific sources has driven the need for automated fact verification systems. Early methods relied on rule-based approaches or static models tailored to specific datasets, limiting scalability. Recent advances in large language models (e.g., GPT-4) have improved natural language understanding but still face challenges in structured data reasoning, especially across multiple databases. Existing systems like BINDER, DATER, and CoTable have made progress in cross-table verification, yet they lack full automation and environment independence. As data sources become more diverse and complex, there is a pressing need for systems that can autonomously explore unknown data landscapes, generate verifiable explanations, and operate seamlessly across heterogeneous environments. Thucy addresses this gap by integrating multi-agent AI with standardized protocols, enabling dynamic discovery and reasoning over multiple relational databases without prior schema knowledge.
Core Problem
Current fact verification methods are predominantly limited to single, well-structured datasets, unable to scale to real-world scenarios involving multiple, heterogeneous databases. Manual schema understanding and query formulation hinder automation and introduce errors. Moreover, existing approaches often lack transparency, making verification results difficult to reproduce or audit. The core challenge is to develop a system that can autonomously discover, interpret, and reason over unknown relational data sources, generate accurate SQL queries, and provide clear, reproducible evidence—all while maintaining high accuracy and interpretability. Achieving this requires overcoming obstacles in schema understanding, query generation, multi-source data integration, and interaction management within a unified framework.
Innovation
Thucy introduces several key innovations:
1) Multi-agent architecture with specialized components—Data Expert, Schema Expert, SQL Expert—coordinated by a Verifier, enabling modular, scalable reasoning.
2) Utilization of the MCP protocol for environment-agnostic database connectivity, allowing seamless integration of diverse data sources.
3) Multi-turn interaction strategy to iteratively refine SQL queries, improving accuracy and robustness.
4) Fully autonomous exploration, from schema discovery to query execution, without prior environment knowledge.
5) Transparent verification process, with detailed SQL evidence supporting each verdict, enhancing interpretability and trustworthiness.
These innovations collectively enable the system to perform complex, cross-database fact verification with minimal human intervention, setting new standards in structured data reasoning.
Methodology
- �� 初始化：系统通过环境无依赖的MCP协议连接多数据库，启动数据专家（Data Expert）进行全局扫描。
- �� 数据探索：数据专家识别所有可用数据库和表，生成简要总结。
- �� 模式理解：模式专家根据用户问题和上下文，深入分析相关数据库结构，回答schema相关问题。
- �� 查询生成：SQL专家根据理解的schema和自然语言问题，生成初步SQL查询。
- �� 多轮优化：SQL专家根据反馈，调整查询，确保语义正确和效率最优。
- �� 证据验证：验证器协调专家，逐步验证SQL查询结果，形成验证结论。
- �� 结果输出：最终验证结果（支持或反驳）和详细SQL证据链被输出，确保过程可追溯。
Experiments
在TabFact数据集上，Thucy系统加载官方结构化数据，执行多项验证任务，采用准确率作为主要指标。对比基线模型，验证其跨数据库验证能力。通过消融实验验证多轮交互和工具集设计的贡献。参数调优包括模型温度、交互轮次等，确保系统稳定性。结果显示，Thucy在复杂多源验证中表现优异，准确率达94.3%，显著优于现有方法。系统还在不同数据库环境下测试其适应性，验证其扩展性和鲁棒性。
Results
Thucy在TabFact上的验证准确率达94.3%，比之前的最优模型高出5.6个百分点。它能自动发现多数据库中的相关表和数据，生成精确SQL查询，支持复杂多表、多源验证。提供详细的SQL证据链，增强验证的透明性。多轮交互优化显著提升SQL生成质量，减少错误。实验证明系统在不同数据库环境下具有良好的适应性和扩展性，展现出强大的实用潜力。
Applications
Thucy适用于公共数据核查、政策评估、舆情监测等场景。用户只需提供结构化数据，系统即可自动验证声明。对政府、媒体、研究机构等具有重要价值，能大幅提升验证效率和可信度。未来结合自动数据采集和多源融合技术，支持非关系型数据库和实时数据流，推动智能验证在实际行业中的应用，助力建立可信信息生态。
Limitations & Outlook
系统依赖高质量、结构合理的数据库，面对噪声或数据不完整时性能下降。多数据库环境下复杂查询可能导致推理耗时较长，实时性不足。当前主要验证在TabFact，实际应用需适应多样化行业场景和数据库结构。未来需优化推理效率，增强对异构数据的支持，提升鲁棒性和扩展性。
Plain Language Accessible to non-experts
想象你在厨房准备一顿大餐，手边有许多不同的食材和厨具。每次做菜前，你要先找到所有的食材，了解它们的特性，然后用合适的厨具把它们组合起来。Thucy就像这个厨房助手，它可以自己探索所有的食材（数据库），理解它们的性质（结构），然后用正确的厨具（SQL）做出一道符合要求的菜（验证结果）。它还能告诉你用的哪些食材和工具，确保每一步都清楚明了。这样一来，你就不用自己翻查所有资料，就能快速、准确地确认菜是不是符合标准，既省时又放心。
ELI14 Explained like you're 14
想象你在学校里听老师说某个学生去年得了多少奖，你和朋友们都想知道是真是假。可是，要确认这个消息，你得问很多老师、查很多资料，还得理解每份资料的内容。Thucy就像一个超级聪明的助手，它可以自己跑去找所有的资料，理解每份资料的内容，然后用一个特别的“翻译器”把问题变成数据库能理解的语言（SQL），再去查证。它会告诉你答案是不是对的，还会把用到的查询语句告诉你，让你知道它是怎么得出结论的。这样一来，你不用自己翻查所有资料，就能快速、准确地知道消息的真假，还能理解它是怎么验证的，既省时又可靠。
Glossary
Multi-Agent System (多智能体系统)
由多个专门的智能体组成，各自负责不同任务，通过协作完成复杂任务。本文中，系统包括数据专家、模式专家和SQL专家。
用以实现跨数据库的自主探索和推理。
MCP协议 (Model Context Protocol)
一种标准协议，用于不同AI工具和模型之间的连接，简化多源数据访问和操作流程。本文利用其实现数据库工具的无缝集成。
支持Thucy中多数据库的工具管理。
SQL专家 (SQL Expert)
负责将自然语言问题转化为SQL查询，并执行以获得验证证据。确保每个查询都能追溯到数据源。
实现自动化、多轮交互的核心组件。
TabFact数据集
用于结构化数据事实验证的标准基准数据集，包含大量已标注的事实验证任务。
评估Thucy在结构化数据验证中的性能。
Open Questions Unanswered questions from this research
- 1 如何支持异构数据源（如非关系型数据库）以及实时数据流，仍需解决多源融合和推理效率问题。
- 2 在复杂行业场景中，确保验证鲁棒性和准确性仍是未来研究重点。
Applications
Immediate Applications
公共数据核查
政府和媒体可用Thucy自动验证政策声明、新闻报道的真实性，提升信息可信度。
政策评估
通过自动分析多源统计数据，帮助决策者快速评估政策效果，减少人为偏差。
Long-term Vision
智能信息验证平台
未来将构建全自动、多源、多行业的验证平台，实现实时、全自动的事实核查，推动可信信息生态。
Abstract
In today's age, it is becoming increasingly difficult to decipher truth from lies. Every day, politicians, media outlets, and public figures make conflicting claims -- often about topics that can, in principle, be verified against structured data. For instance, statements about crime rates, economic growth or healthcare can all be verified against official public records and structured datasets. Building a system that can automatically do that would have sounded like science fiction just a few years ago. Yet, with the extraordinary progress in LLMs and agentic AI, this is now within reach. Still, there remains a striking gap between what is technically possible and what is being demonstrated by recent work. Most existing verification systems operate only on small, single-table databases -- typically a few hundred rows -- that conveniently fit within an LLM's context window. In this paper we report our progress on Thucy, the first cross-database, cross-table multi-agent claim verification system that also provides concrete evidence for each verification verdict. Thucy remains completely agnostic to the underlying data sources before deployment and must therefore autonomously discover, inspect, and reason over all available relational databases to verify claims. Importantly, Thucy also reports the exact SQL queries that support its verdict (whether the claim is accurate or not) offering full transparency to expert users familiar with SQL. When evaluated on the TabFact dataset -- the standard benchmark for fact verification over structured data -- Thucy surpasses the previous state of the art by 5.6 percentage points in accuracy (94.3% vs. 88.7%).