DecAF: Joint Decoding of Answers and Logical Forms for Question Answering over Knowledge Bases
DecAF jointly decodes logical forms and direct answers using text retrieval, achieving SOTA on WebQSP, FreebaseQA, GrailQA with 79-82% Hits@1.
Key Findings
Methodology
DecAF employs a multi-task sequence-to-sequence framework that generates logical forms and direct answers simultaneously, guided by distinct prefixes. It linearizes large knowledge bases into text documents, then applies BM25 or DPR for passage retrieval. The retrieved passages, combined with the question, are encoded via FiD, enabling cross-passage reasoning. During decoding, the model produces both logical forms and answers, which are executed and fused through a weighted linear combination, optimizing final answer accuracy. This approach leverages the strengths of semantic parsing and direct answer prediction while simplifying the pipeline by removing entity linking dependencies.
Key Results
- On WebQSP, FreebaseQA, and GrailQA, DecAF surpasses previous SOTA with Hits@1 scores of 79.0%, 80.7%, and 78.8%, respectively. It demonstrates significant improvements in complex multi-hop reasoning and zero-shot scenarios. Ablation studies show that joint decoding and knowledge base linearization contribute substantially to performance gains. The model maintains robustness across different retrieval strategies (BM25, DPR) and model sizes (FiD-large, FiD-3B).
- In GrailQA, DecAF achieves the highest overall F1 (78.8%) and excels particularly in compositional questions, outperforming existing methods by over 5%. The logical form generation success rate exceeds 88%, with non-executable logical forms reduced to 11.3%. These results confirm the effectiveness of the joint decoding approach in diverse question types.
- Replacing entity linking with text retrieval simplifies the pipeline and enhances cross-domain adaptability. The experiments validate that a straightforward retrieval-based knowledge linearization combined with multi-task decoding yields competitive, if not superior, results compared to more complex, entity-dependent systems.
Significance
This work advances KBQA by integrating logical reasoning and direct answer prediction into a unified, end-to-end framework. It addresses longstanding issues of logical form non-executability and domain dependence, offering a scalable, adaptable solution. The elimination of entity linking reliance broadens applicability across diverse datasets and knowledge bases. The approach paves the way for more robust, interpretable, and scalable question answering systems, with potential impacts on virtual assistants, enterprise search, and AI-driven knowledge discovery.
Technical Contribution
DecAF introduces a novel joint decoding paradigm, where a shared sequence-to-sequence model generates logical forms and answers conditioned on different prompts. It innovatively linearizes knowledge bases into text, enabling retrieval-based knowledge selection without entity linking. The model employs FiD for multi-passage encoding, facilitating cross-passage reasoning. The fusion of logical form execution results with directly generated answers via weighted scoring enhances answer reliability. These contributions collectively push the frontier of scalable, flexible KBQA models.
Novelty
This is the first work to unify logical form generation and direct answer prediction within a single, joint decoding framework guided by different prefixes. It replaces traditional entity linking with knowledge base linearization and text retrieval, simplifying the pipeline and improving generalization. Unlike prior methods relying heavily on constrained decoding or external executors, DecAF leverages standard text generation, making it more adaptable and easier to extend to new datasets and KBs.
Limitations
- Despite improvements, approximately 11.3% of logical forms remain non-executable, affecting overall answer completeness. Further refinement of logical form generation is needed.
- Retrieval efficiency could become a bottleneck with extremely large knowledge bases, especially in real-time applications.
- Training large models like FiD-3B requires substantial computational resources, limiting accessibility for some users.
- Current approach may struggle with very noisy or incomplete knowledge bases, requiring robustness enhancements.
Future Work
Future directions include optimizing logical form generation to reduce non-executable outputs, integrating more intelligent retrieval mechanisms, and exploring multi-modal data sources. Extending the framework to multilingual settings and low-resource domains is also promising. Additionally, incorporating user feedback for iterative refinement could further improve answer quality and system robustness.
AI Executive Summary
Knowledge base question answering (KBQA) has long grappled with balancing the accuracy of logical reasoning and the robustness of direct answer prediction. Traditional semantic parsing approaches generate logical forms that, while precise, often suffer from non-executability issues due to syntactic or semantic errors. Conversely, direct answer prediction models are more robust but less capable of complex multi-hop reasoning. This dichotomy has limited the overall performance and scalability of KBQA systems.
The authors introduce DecAF, a novel framework that unifies these two paradigms through joint decoding. By guiding a shared sequence-to-sequence model with distinct prefixes, DecAF simultaneously generates logical forms and direct answers, leveraging the strengths of both. The key innovation lies in linearizing large-scale knowledge bases into text documents, which are then retrieved via BM25 or DPR methods, bypassing the need for entity linking. This simplification enhances the model’s adaptability across datasets and KBs.
At the core of DecAF is the Fusion-in-Decoder (FiD) architecture, which encodes multiple retrieved passages for cross-passage reasoning. During inference, the model produces both logical forms and answers, which are executed and combined through a weighted scoring mechanism. This process ensures that the final answer benefits from logical reasoning’s precision and direct prediction’s robustness.
Extensive experiments on WebQSP, FreebaseQA, GrailQA, and CWQ demonstrate that DecAF achieves state-of-the-art results, with Hits@1 reaching up to 82.1% on WebQSP and 78.8% on GrailQA. Notably, it performs well in complex, multi-hop, and zero-shot scenarios, validating its generalization capabilities. The approach’s simplicity, avoiding entity linking and relying on text retrieval, makes it highly scalable and adaptable.
This work significantly advances KBQA by providing a unified, efficient, and high-performing solution. It opens avenues for further research into multi-task joint decoding, improved logical form generation, and broader application across diverse knowledge sources. Despite some remaining challenges, such as logical form non-executability and retrieval efficiency, DecAF sets a new benchmark for scalable, accurate question answering over large knowledge bases.
Deep Dive
Glossary
Knowledge Base Linearization (知识库线性化)
将结构化知识库内容转化为连续文本的过程，便于文本检索和匹配。In this paper, it refers to converting triplets into sentences for retrieval.
用于简化知识库的检索流程，避免实体链接依赖。
Fusion-in-Decoder (FiD)
一种多通道编码架构，将多个检索段落编码后在解码时融合，增强多跳推理能力。In this paper,用于处理多段检索信息。
实现跨段落的交叉推理，提升复杂问答表现。
Joint Decoding (联合解码)
同时生成多种输出（如逻辑形式和答案）的模型策略。In this paper,通过不同前缀引导，实现逻辑形式和答案的同步生成。
融合推理与直答优势，提升整体性能。
Logical Form (逻辑形式)
表达知识库查询的结构化语句（如SPARQL），用于推理。In this paper,作为文本串生成，无需实体ID。
实现推理的可执行性。
DPR (Dense Passage Retrieval)
基于BERT的密集向量空间检索模型，用于高效匹配相关段落。In this paper,替代BM25进行知识检索。
提升检索相关性和鲁棒性。
Open Questions Unanswered questions from this research
- 1 如何进一步降低逻辑形式非可执行比例，提升推理准确性？
- 2 在极大规模知识库中，检索效率与准确性如何平衡？
- 3 模型训练成本高，如何实现更高效的训练与部署？
Applications
Immediate Applications
企业知识问答系统
利用DecAF实现企业内部知识库的快速问答，减少人工维护成本，提升信息检索效率。
智能助手升级
集成DecAF于智能助手中，增强复杂问题的理解与回答能力，改善用户体验。
Long-term Vision
多模态知识问答
结合图像、视频等多模态信息，扩展知识表达与推理能力，推动智能系统的多源信息整合。
Abstract
Question answering over knowledge bases (KBs) aims to answer natural language questions with factual information such as entities and relations in KBs. Previous methods either generate logical forms that can be executed over KBs to obtain final answers or predict answers directly. Empirical results show that the former often produces more accurate answers, but it suffers from non-execution issues due to potential syntactic and semantic errors in the generated logical forms. In this work, we propose a novel framework DecAF that jointly generates both logical forms and direct answers, and then combines the merits of them to get the final answers. Moreover, different from most of the previous methods, DecAF is based on simple free-text retrieval without relying on any entity linking tools -- this simplification eases its adaptation to different datasets. DecAF achieves new state-of-the-art accuracy on WebQSP, FreebaseQA, and GrailQA benchmarks, while getting competitive results on the ComplexWebQuestions benchmark.