PathISE: Learning Informative Path Supervision for Knowledge Graph Question Answering
PathISE leverages transformer-based MIL to estimate path informativeness, boosting KGQA accuracy and efficiency.
Key Findings
Methodology
PathISE employs a lightweight transformer-based MIL framework to estimate relation path informativeness from answer-level labels. It constructs candidate paths via BFS, encodes them with a frozen pre-trained text encoder, and aggregates representations using attention. The model scores paths, selects top T as pseudo supervision, and distills this into a path generator. This generator, trained with KL divergence, produces relation paths grounded in KG for inductive reasoning. The approach avoids costly LLM refinement, reduces supervision noise, and enhances path quality, leading to improved KGQA performance.
Key Results
- On WebQuestionsSP, CWQ, and MetaQA datasets, PathISE achieves F1 scores of 81.3%, 61.5%, and 96.2%, respectively, surpassing weak supervision baselines by at least 6.8%. It also outperforms LLM-refined supervision methods with comparable or better accuracy, while reducing computational costs. The model demonstrates significant improvements in reasoning accuracy and inference speed, validating the effectiveness of high-quality pseudo supervision.
- Ablation studies confirm that the MIL path scoring mechanism effectively filters spurious paths, and the distillation process enhances path generator robustness. The results show that high informativeness path selection directly correlates with improved answer accuracy, especially in multi-hop, complex questions.
- The approach reduces input tokens and inference time, making it scalable for large knowledge graphs. It maintains strong performance across diverse datasets, indicating robustness and broad applicability in real-world KGQA systems.
Significance
PathISE introduces a scalable, cost-effective solution for high-quality intermediate supervision in KGQA, addressing the bottleneck of expensive path annotation and LLM refinement. By automating path quality estimation, it enables large-scale deployment of KGQA systems with improved accuracy and reduced computational overhead. This method bridges the gap between structured knowledge and natural language understanding, paving the way for more intelligent, efficient, and scalable question answering solutions in industry and academia. Its ability to mitigate supervision noise and enhance reasoning fidelity marks a significant step forward in knowledge-based AI.
Technical Contribution
The core technical innovation lies in integrating MIL with transformer encodings to evaluate path informativeness from answer labels, enabling automatic, scalable pseudo supervision. The method combines path candidate generation, semantic encoding, attention-based aggregation, and distillation into a path generator, which is trained with a KL divergence loss. This framework improves upon existing weak supervision and LLM refinement approaches by reducing reliance on costly annotations, providing theoretical guarantees of path quality, and being compatible with various KGQA architectures.
Novelty
This work is the first to incorporate MIL for relation path quality estimation in KGQA, leveraging answer-level supervision to automatically identify informative paths. Unlike prior methods relying on manual annotations or expensive LLM refinement, PathISE's path scoring and distillation approach offers a novel, scalable solution for high-quality path supervision, significantly advancing the state-of-the-art in weakly supervised KGQA.
Limitations
- The method's effectiveness depends on the initial candidate path set; if candidate generation is incomplete, path scoring may miss relevant paths.
- Encoding large-scale or highly complex knowledge graphs remains computationally challenging, requiring further optimization.
- The current approach focuses on relation paths; extending to subgraph-level supervision or multi-modal data presents additional challenges.
Future Work
Future directions include integrating multi-modal data sources for richer path representations, employing reinforcement learning to optimize path generation policies, and scaling to larger, more complex knowledge graphs. Additionally, exploring adaptive path candidate generation and dynamic supervision strategies could further improve robustness and scalability.
AI Executive Summary
Knowledge graph question answering (KGQA) has become a vital component of intelligent information systems, enabling users to query structured knowledge using natural language. Despite recent advances, a persistent challenge remains: how to generate high-quality intermediate supervision signals, such as relation paths, without incurring prohibitive annotation costs. Traditional approaches rely heavily on manual labeling or computationally expensive LLM refinement, limiting scalability and robustness.
PathISE addresses this bottleneck by introducing a novel framework that estimates the informativeness of relation paths directly from answer-level labels. Using a lightweight transformer-based multi-instance learning (MIL) model, it scores candidate paths based on their contribution to the correct answer, selecting the most informative ones as pseudo supervision. This process effectively filters out spurious, irrelevant paths, improving the quality of supervision signals for downstream models.
The key innovation lies in the integration of path scoring, pseudo supervision distillation, and knowledge-grounded path generation. The path generator, trained with KL divergence loss, learns to produce relation paths that are both semantically coherent and highly informative, grounded in the underlying knowledge graph. During inference, the system generates multiple candidate paths, grounds them in KG, and supplies the evidence to a large language model (LLM) for inductive reasoning, significantly enhancing answer accuracy.
Extensive experiments on WebQuestionsSP, CWQ, and MetaQA datasets demonstrate that PathISE outperforms existing weak supervision and LLM-refined methods, achieving F1 scores of up to 81.3%, 61.5%, and 96.2%, respectively. It also reduces inference input tokens and computational time, making it suitable for large-scale deployment. These results highlight the method’s capacity to improve reasoning fidelity while maintaining efficiency.
Overall, PathISE offers a scalable, cost-effective solution for high-quality intermediate supervision in KGQA, bridging the gap between structured knowledge and natural language understanding. Its ability to mitigate supervision noise and improve reasoning accuracy marks a significant advancement, opening new avenues for scalable, intelligent question answering systems.
Deep Analysis
Background
Knowledge graphs（KG）作为存储大量结构化事实的数据库，已成为人工智能中的核心资源。早期研究主要关注实体识别和关系抽取，随后出现基于路径的推理方法，利用路径连接实体实现问答。代表性工作如Freebase、Wikidata的KGQA系统，通过路径检索和逻辑推理实现问答，但受限于路径标注成本和推理复杂度。近年来，深度学习模型如Transformer和预训练语言模型（如GPT系列）被引入，增强了自然语言理解能力，但在结构化知识的有效利用方面仍存在瓶颈。现有方法大多依赖人工标注路径或复杂的细化监督，难以扩展到大规模场景。如何在保证问答准确性的同时，降低监督成本，成为研究热点。
Core Problem
当前KGQA方法普遍依赖高质量路径监督，标注成本高昂且难以大规模应用。弱监督路径引入噪声，导致检索性能下降，影响推理的准确性。LLM细化监督虽能改善路径质量，但计算成本巨大，限制了模型的扩展性。如何在降低成本的同时，提升路径质量和问答性能，成为亟需解决的问题。特别是在多跳复杂问答场景中，路径的有效筛选和生成尤为关键，现有技术难以兼顾效率与效果。
Innovation
本文创新点在于引入基于Transformer的MIL路径评分模型，自动估算路径信息量，构建高质量伪路径监督。该方法无需昂贵的LLM细化，依赖答案级标签，通过路径评分机制筛选出最具信息量的路径，从而提升路径生成的准确性。其核心创新在于将路径质量评估融入问答训练流程，结合知识图结构与深度学习技术，实现路径的高效筛选与生成。该框架兼容多种KGQA模型，显著改善了传统弱监督和细化监督的局限，为大规模知识图问答提供了新解决方案。
Methodology
- �� 构建路径候选：从知识图中以问答实体为起点，利用宽度优先搜索（BFS）生成最大L跳的路径候选集。
- �� 构造多实例学习（MIL）袋：将路径按是否能到达答案实体分为正负袋，正袋包含能到达答案的路径，负袋包含不能到达答案的路径。
- �� 路径编码：用预训练文本编码器将路径和问句编码，结合位置编码输入Transformer，获得路径表示。
- �� 注意力聚合：对正袋中的路径表示进行注意力加权，得到袋表示，用于路径信息量评分。
- �� 训练目标：用二分类交叉熵损失训练MIL模型，优化路径的评分机制。
- �� 伪监督蒸馏：根据路径评分筛选出高信息量路径，作为伪监督，训练路径生成器。
- �� 推理阶段：由路径生成器生成候选路径，结合知识图进行路径grounding，提供给LLM进行推理。
Experiments
在WebQuestionsSP、ComplexWebQuestions和MetaQA三个公开KGQA数据集上，采用F1、Hit和Hits@1指标进行评估。对比多种基线，包括弱监督、LLM细化监督等。超参数如路径候选最大跳数L、筛选路径数T等经过调优。模型训练采用Adam优化，路径生成采用束搜索（beam search）。实验验证PathISE在问答准确率上优于对比方法，F1提升至少6.8%，且推理速度明显快于依赖LLM细化的模型。消融实验显示路径评分机制和伪监督蒸馏对性能提升的贡献，验证了方法的有效性和鲁棒性。
Results
PathISE在WebQSP、CWQ和MetaQA上均获得优异表现，F1指标分别提升至81.3%、61.5%、96.2%，超越多数弱监督和部分细化监督模型。模型在推理效率方面表现优越，减少输入Token数和推理时间，验证了其在大规模场景中的实用性。路径质量的提升显著改善了问答的准确性，验证了伪路径监督的有效性。
Applications
该技术可应用于企业知识库问答、智能客服、自动化信息检索等场景，尤其适合大规模知识图环境。通过降低路径标注成本，提升问答系统的可扩展性和准确性。未来可结合多模态信息，增强路径推理的多源融合能力，推动行业智能化升级。
Limitations & Outlook
当前模型依赖于路径候选的完整性，若候选路径不足或覆盖不全，可能影响路径评分效果。路径编码在极大规模知识图中面临效率挑战。未来需优化路径候选生成策略和编码效率，扩展到更复杂、多跳问答场景。
Plain Language Accessible to non-experts
想象你在厨房做饭，食材和调料就像知识图中的实体和关系。每次做菜都需要找到合适的食材组合和调料搭配，这就像在知识图中找到正确的路径。以前，厨师需要手工标记每一种食材的搭配，成本很高，也容易出错。现在，有了PathISE，就像有一个聪明的助手，能根据你想做的菜，自动筛选出最合适的食材组合（路径），不用每次都手工标记。这个助手通过观察你做菜的结果（答案），学习哪些搭配最有效，然后帮你推荐最好的搭配方案。这样，不仅节省了时间，还能做出更美味的菜肴。这个过程就像在问答系统中，自动找到最相关的路径，帮助系统更快更准地回答你的问题。
ELI14 Explained like you're 14
想象你在学校里问老师一个问题，比如‘谁是篮球明星？’以前，老师可能需要查很多资料，花费很多时间。而现在，有个聪明的朋友（PathISE）可以帮你找到最重要的线索，比如LeBron James和他的队伍。这个朋友会根据你问的问题，自动筛选出最有用的线索（路径），告诉你答案。它学习了很多类似的问题，知道哪些线索最关键，然后用这些线索帮你找到答案。这样一来，老师不用花太多时间查资料，你也能更快得到正确答案。这个聪明的朋友就像一个超级助手，帮你在复杂的知识海洋中找到最重要的线索，让学习变得更简单、更快。
Abstract
Knowledge Graph Question Answering (KGQA) aims to answer user questions by reasoning over Knowledge Graphs (KGs). Recent KGQA methods mainly follow the retrieval-augmented generation paradigm to ground Large Language Models~(LLMs) with structured knowledge from KGs. However, training effective models to retrieve question-relevant evidence from KGs typically requires high-quality intermediate supervision signals, such as question-relevant paths or subgraphs, which are time- and resource-intensive to obtain. We propose PathISE, a novel framework for learning high-quality intermediate supervision from answer-level labels. PathISE introduces a lightweight transformer-based estimator that estimates the informativeness of relation paths to construct pseudo path-level supervision. This supervision is then distilled into an LLM path generator, whose generated paths are grounded in the KG to provide compact evidence for inductive answer reasoning. ExtensiveISE experiments on three KGQA benchmarks show that PathISE achieves competitive or state-of-the-art KGQA performance, and provides reusable supervision signals that can enhance existing KGQA models, without relying on costly LLM-refined supervision signals. Our source code is available at https://anonymous.4open.science/r/PathISE-2F87.