QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering
QA-GNN combines relevance scoring and joint graph reasoning, outperforming LM+KG models with 4.7% improvement on CommonsenseQA.
Key Findings
Methodology
QA-GNN integrates pre-trained language models (e.g., RoBERTa) with knowledge graphs (e.g., ConceptNet, UMLS). It first computes relevance scores for KG nodes conditioned on the QA context, filtering noisy or irrelevant entities. Then, it constructs a joint graph with a special QA context node connected to topic entities, enabling multi-layer graph attention networks (GAT) to perform iterative message passing. This process updates both textual and knowledge representations simultaneously, facilitating structured reasoning. End-to-end training aligns semantic and structural cues, boosting inference accuracy and interpretability.
Key Results
- On datasets like CommonsenseQA, OpenBookQA, and MedQA-USMLE, QA-GNN surpasses the best LM+KG baselines by 4.7%, 2.3%, and 1.2% respectively. It notably improves performance on questions involving negation and entity substitution, demonstrating robustness in complex reasoning scenarios.
- In CommonsenseQA, the model achieves a 4.6% accuracy gain on negation questions, outperforming previous models by 0.6%. On MedQA-USMLE, it reaches 38.0%, validating cross-domain applicability. The model also extracts multiple reasoning paths, enhancing explainability.
- Ablation studies confirm that relevance scoring and joint graph reasoning significantly contribute to performance gains, especially in structured reasoning tasks.
Significance
This work advances the integration of semantic and structural knowledge, addressing the limitations of static subgraph retrieval and path-based reasoning. By dynamically filtering relevant knowledge and enabling multi-hop reasoning, QA-GNN enhances both accuracy and interpretability. Its framework is adaptable across domains, paving the way for more intelligent, explainable AI systems capable of complex logical inference, crucial for applications like medical diagnosis, scientific question answering, and commonsense reasoning.
Technical Contribution
The core innovation lies in leveraging pre-trained LM-based relevance scores to dynamically select KG nodes, combined with a multi-layer GAT that performs joint reasoning over a unified graph. This end-to-end architecture effectively fuses textual semantics with structured knowledge, surpassing prior static or path-based methods. The relevance scoring mechanism introduces a flexible, learnable importance measure, while the joint graph enables deep, multi-hop reasoning with interpretability. The approach also scales efficiently compared to relation-specific RGCN variants.
Novelty
This is the first systematic integration of LM-based relevance scoring with joint graph reasoning for question answering. Unlike previous models that rely on static subgraph extraction or path enumeration, QA-GNN dynamically filters knowledge, enabling more precise and flexible reasoning. Its end-to-end design and multi-hop attention mechanism set a new standard for knowledge-aware NLP models, especially in handling negation and complex logical structures.
Limitations
- The model heavily depends on the quality of pre-trained LM relevance scores; inaccuracies here can mislead the reasoning process. Handling extremely large knowledge graphs remains computationally intensive, requiring further optimization.
- Generalization to open-domain, multi-modal, or multi-task scenarios needs validation, as current experiments focus on specific datasets. The interpretability, while improved, still faces challenges in explaining multi-hop paths comprehensively.
- Training complexity and resource demands are high, limiting deployment in resource-constrained environments. Future work should explore model compression and efficiency improvements.
Future Work
Future directions include integrating multi-modal data (images, audio), enhancing scalability to larger knowledge bases, and improving interpretability through more transparent reasoning paths. Exploring unsupervised relevance scoring and transfer learning across tasks could further broaden applicability. Additionally, optimizing computational efficiency for real-time deployment remains a key goal.
AI Executive Summary
Answering complex questions requires not only understanding language but also leveraging structured knowledge. Traditional models either rely solely on large language models (LMs) like RoBERTa, which excel at language understanding but struggle with structured reasoning, or on knowledge graphs (KGs) like ConceptNet, which provide explicit relational information but lack dynamic relevance filtering. Combining these modalities has been promising but challenging, often limited by static subgraph extraction and path-based reasoning that cannot adapt to nuanced contexts.
This paper introduces QA-GNN, a novel framework that bridges the gap by dynamically assessing the relevance of KG nodes conditioned on the question context, and then performing joint reasoning over a unified graph structure. The key innovation is the relevance scoring mechanism, which uses a pre-trained LM to evaluate the importance of each entity in the KG, filtering out noise and irrelevant information. This relevance score is then incorporated into a joint graph that includes a special QA context node connected to topic entities, enabling multi-layer graph attention networks (GAT) to propagate information iteratively.
The entire system is trained end-to-end, allowing the model to learn optimal relevance assessments and reasoning strategies simultaneously. Extensive experiments on datasets like CommonsenseQA, OpenBookQA, and MedQA-USMLE demonstrate that QA-GNN outperforms existing LM and LM+KG baselines, with improvements of up to 4.7% in accuracy. Notably, it handles negation and entity substitution more effectively, showing robustness in structured reasoning tasks. The model also provides interpretable reasoning paths, enhancing transparency.
Overall, QA-GNN represents a significant step forward in knowledge-based question answering, combining the semantic richness of pre-trained LMs with the explicit relational structure of KGs. Its flexible, scalable architecture opens new avenues for AI systems capable of complex, explainable reasoning across diverse domains. Future work will focus on multi-modal integration, efficiency improvements, and broader applicability in real-world scenarios.
Deep Analysis
Background
The evolution of question answering (QA) systems has seen significant advances with the advent of large pre-trained language models (LMs) like BERT and RoBERTa, which excel at understanding natural language semantics. Concurrently, structured knowledge graphs (KGs) such as ConceptNet, Freebase, and UMLS have provided explicit relational data that supports logical reasoning. Early approaches integrated these sources via static subgraph retrieval or path enumeration, but faced limitations in scalability, relevance filtering, and handling complex logical constructs like negation. Recent efforts, including Lin et al. (2019) and Feng et al. (2020), attempted to fuse LM and KG information through relation-aware GNNs or path-based reasoning, yet struggled with noisy knowledge, fixed subgraph scopes, and interpretability. The challenge remains to dynamically select relevant knowledge and perform deep, flexible reasoning that can handle nuanced linguistic phenomena.
Core Problem
The core challenge addressed is how to efficiently identify and utilize the most relevant knowledge from large-scale KGs for each QA instance, while enabling the model to perform structured, multi-hop reasoning that captures logical nuances such as negation or entity substitution. Static subgraph extraction often introduces irrelevant nodes, reducing reasoning accuracy and interpretability. Existing models lack dynamic relevance assessment and joint reasoning capabilities, limiting their ability to adapt to diverse question types and complex logical structures. Overcoming these bottlenecks is crucial for advancing AI's reasoning and explainability in real-world applications.
Innovation
The paper introduces two main innovations: 1) Relevance scoring via a pre-trained LM, which evaluates the importance of each KG node conditioned on the QA context, enabling dynamic filtering and weighting of knowledge. 2) Construction of a joint graph that incorporates a dedicated QA context node connected to topic entities, facilitating multi-layer GAT-based message passing. This design allows simultaneous updating of textual and knowledge representations, capturing complex relations and negations more effectively than prior path-based or static subgraph methods. The end-to-end training ensures optimal fusion of semantic and structural cues, setting a new standard for knowledge-aware QA.
Methodology
- �� Encode QA context with a pre-trained LM (e.g., RoBERTa) to obtain a contextual vector.
- �� Retrieve a subgraph from the KG based on topic entities, then prune to top 200 nodes using relevance scores.
- �� Compute relevance scores for each node by concatenating entity text with QA context and passing through the LM.
- �� Construct a joint graph by adding a QA context node connected to topic entities with relation types rz,q and rz,a.
- �� Initialize node embeddings with LM representations and entity embeddings.
- �� Perform multi-layer GAT message passing, incorporating type, relation, and relevance-aware attention mechanisms.
- �� Aggregate node representations for final answer prediction via a classification layer.
Experiments
The evaluation used CommonsenseQA, OpenBookQA, and MedQA-USMLE datasets, comparing against baselines like fine-tuned LM, RGCN, KagNet, and MHGRN. Hyperparameters included 5 GNN layers, 200-dimensional embeddings, and specific learning rates. Ablation studies tested relevance scoring and joint reasoning effects. Metrics focused on accuracy, with statistical significance assessed through multiple runs. The model was trained end-to-end with cross-entropy loss, ensuring joint optimization of relevance estimation and reasoning modules.
Results
QA-GNN achieved 73.41% on CommonsenseQA, surpassing MHGRN by 2.3%. On OpenBookQA, it reached 82.8%, outperforming previous models. In the biomedical domain, it scored 38.0% on MedQA-USMLE, outperforming models like SapBERT alone. The model excelled particularly in questions involving negation, entity substitution, and multi-hop reasoning, demonstrating robustness and interpretability. Ablation results confirmed that relevance scoring and joint graph reasoning contributed significantly to performance improvements.
Abstract
The problem of answering questions using knowledge from pre-trained language models (LMs) and knowledge graphs (KGs) presents two challenges: given a QA context (question and answer choice), methods need to (i) identify relevant knowledge from large KGs, and (ii) perform joint reasoning over the QA context and KG. In this work, we propose a new model, QA-GNN, which addresses the above challenges through two key innovations: (i) relevance scoring, where we use LMs to estimate the importance of KG nodes relative to the given QA context, and (ii) joint reasoning, where we connect the QA context and KG to form a joint graph, and mutually update their representations through graph neural networks. We evaluate our model on QA benchmarks in the commonsense (CommonsenseQA, OpenBookQA) and biomedical (MedQA-USMLE) domains. QA-GNN outperforms existing LM and LM+KG models, and exhibits capabilities to perform interpretable and structured reasoning, e.g., correctly handling negation in questions.