SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs
SPARK uses knowledge graphs for structured multi-hop reasoning, outperforming unstructured baselines with 93% accuracy on ScienceQA.
Key Findings
Methodology
SPARK constructs a three-stage multimodal knowledge graph (structural, reference, semantic) from scientific papers. A single vision-language model (sVLM) alternates roles in a self-play loop, sampling KG paths to generate relational questions and answering without KG access. Rewards are decomposed into answer correctness, path faithfulness, and factual consistency, verified against the KG. This setup enables the model to learn deep relational reasoning, especially in multi-hop (up to 3-hop) scenarios, outperforming flat corpus baselines. The framework incorporates continuous KG refinement through edge detection and pruning, fostering co-evolution of model and knowledge structure.
Key Results
- On ScienceQA, SPARK achieves 93.0% accuracy, surpassing baseline models like Qwen3-VL-4B (90.14%). In cross-document multi-hop QA, 3-hop accuracy reaches 60.32%, a significant improvement over non-structured methods. Ablation shows path faithfulness (Rpath) contributes most to performance, validating the importance of structural reasoning.
- Results demonstrate that as hop count increases, SPARK's performance degrades more slowly than unstructured models, confirming the effectiveness of KG grounding. Path F1 scores align with accuracy improvements, indicating reasoning paths follow the KG structure. Reward component analysis highlights the synergy of answer correctness, path faithfulness, and factual consistency.
- The iterative KG refinement process enhances reasoning capabilities, with edge detection identifying implicit relations and pruning removing unreliable edges, leading to a robust knowledge structure that supports complex reasoning.
Significance
This work advances the frontier of scientific literature understanding by embedding explicit relational structures into self-supervised learning. The use of knowledge graphs as a backbone for reasoning and reward verification addresses longstanding challenges of implicit relations and unverified answers in automated scientific QA. It paves the way for more reliable, interpretable, and scalable AI systems capable of deep multi-hop reasoning across diverse scientific domains, with broad implications for automated literature synthesis, research assistance, and knowledge management.
Technical Contribution
The main technical innovation lies in integrating a three-stage multimodal KG construction pipeline with a self-play reinforcement learning framework, where a single sVLM alternates roles under information asymmetry. The path sampling mechanism, combined with structure-aware rewards (path faithfulness and factual consistency), ensures the model learns deep relational reasoning. The dynamic KG refinement process, including edge detection and pruning, creates a co-evolutionary loop that enhances both knowledge quality and reasoning ability. This approach bridges the gap between unstructured text and structured reasoning, offering a scalable, verifiable, and interpretable training paradigm.
Novelty
This is the first work to embed knowledge graph-based structure explicitly into a self-play reinforcement learning framework for scientific literature understanding. Unlike prior methods relying solely on unstructured corpora, SPARK leverages explicit multi-hop KG paths for question generation and reward verification. Its dynamic KG construction and continuous refinement enable scalable, deep relational reasoning across multi-modal, cross-document content, setting a new standard for AI-driven scientific knowledge synthesis.
Limitations
- The knowledge graph construction depends heavily on rule-based parsing and pretrained models, which may struggle with complex or novel content, leading to incomplete or noisy graphs.
- Training costs are high due to iterative KG refinement and large model parameters; scalability to broader domains remains challenging.
- Current approach focuses on scientific literature; extending to other domains requires domain-specific adaptations and larger datasets.
Future Work
Future research will explore online, real-time knowledge graph updates, integrating larger foundation models for enhanced reasoning depth, and expanding to broader scientific fields. Developing more robust, automated KG construction techniques and improving multi-modal fusion efficiency will further enhance the framework's scalability and applicability in real-world scientific AI systems.
AI Executive Summary
Understanding the complex web of scientific knowledge across multiple documents and modalities has long been a challenge for AI systems. Traditional approaches, relying on linear text processing, often fail to capture the rich, multi-layered relationships that underpin scientific reasoning. Recognizing this gap, researchers led by Hyobin Park introduced SPARK, a novel framework that leverages knowledge graphs as the structural backbone for relational reasoning.
SPARK constructs a comprehensive, multimodal knowledge graph through a three-stage pipeline: capturing physical structure, cross-modal references, and semantic relationships. This graph encodes the intricate connections among figures, equations, claims, and concepts across multiple documents. A single vision-language model (sVLM) then alternates roles in a self-play loop, sampling paths from the KG to generate relational questions and answering without direct KG access. Rewards are carefully designed to verify answer correctness, path fidelity, and factual consistency, all grounded in the KG structure.
Experimental results demonstrate that SPARK significantly outperforms baseline models on standard scientific QA benchmarks, achieving 93% accuracy on ScienceQA and improving multi-hop reasoning accuracy by nearly 10% in 3-hop scenarios. The key innovation lies in the path-based reward system, which enforces structural fidelity and factual grounding, enabling the model to learn deep, multi-step relations. The framework's ability to dynamically refine the KG through edge detection and pruning further enhances reasoning robustness.
This work marks a major step toward AI systems capable of understanding and reasoning about complex scientific knowledge. By embedding explicit relational structures and verifying reasoning paths, SPARK addresses fundamental limitations of prior unstructured methods, opening avenues for automated literature synthesis, intelligent research assistants, and scalable knowledge management. Challenges remain in automating KG construction and scaling to broader domains, but the potential impact on scientific discovery and AI interpretability is profound.
Deep Analysis
Background
科学文献理解一直是人工智能领域的重要难题。早期工作如BERT、SciBERT在文本表示方面取得突破，但难以捕获深层关系。知识图谱（KG）作为结构化知识存储工具，已在问答和推理任务中展现潜力，尤其在多跳关系推理（如KGQA）中表现优越。近年来，结合多模态信息的模型（如VisualBERT）推动了科学文献的多模态理解。然而，自动构建跨文档、多模态的知识图谱仍是难点，现有方法多依赖手工标注或规则，难以规模化。自我对弈和强化学习在文本生成和推理中展现潜力，但缺乏结构化知识基础。SPARK正是在此背景下提出，旨在自动化构建多模态知识图谱，并通过结构化奖励实现深层关系推理。
Core Problem
科学文献中的关系隐含且复杂，传统自我对弈方法难以生成高质量的关系推理问题，且奖励信号缺乏结构保障，导致推理深度不足。无结构文本难以验证答案的正确性，也限制了多跳推理的实现。现有方法多依赖手工标注或简单的事实匹配，无法有效捕获跨文档、多模态的复杂关系，限制了模型的推理能力。如何自动构建结构化的知识图谱，支持多模态信息融合，并在自我对弈中实现可验证的奖励，是当前的核心难题。
Innovation
本研究的创新在于：
1）提出三阶段知识图谱构建流程（结构、引用、语义关系），实现多模态内容的自动融合，覆盖文本、图表、公式。
2）引入路径采样机制，生成多跳关系推理题，形成系统性课程。
3）设计结构化奖励（答案正确性、路径忠实度、事实一致性），确保推理路径的结构性和事实基础。
4）采用单一视觉-语言模型（sVLM）在信息不对称下进行角色切换，简化架构同时增强推理能力。
5）实现知识图谱的动态自我修正，边缘检测和修剪机制不断优化知识结构。这一体系突破了无结构文本的限制，为科学文献深度理解提供新范式。
Methodology
- �� 采用三阶段流程：
- 结构图：基于规则解析物理层级关系（章节、段落、图表等）构建基础结构。
- 引用图：利用正则表达式匹配交叉引用，连接不同内容单元。
- 语义关系图：通过VLM分类关系（支持、反对、导出等）补充深层关系。
- �� 跨文档融合：连接不同论文中的相似概念（SAMECONCEPT边），实现跨文档推理路径。
- �� 自我对弈机制：
- Proposer模型采样路径生成关系推理题，基于路径类型和长度调整难度。
- Solver模型回答问题，缺少路径信息，模型在信息不对称中学习。
- �� 奖励设计：
- Ranswer：答案正确性（关键词匹配、数值一致）
- Rpath：路径忠实度（路径是否符合KG关系）
- Rconistency：事实一致性（答案中的关键概念和数值是否与KG事实匹配）
- �� KG自我修正：
- 边缘检测：高奖励路径中新发现关系加入KG。
- 边缘修剪：低置信度边缘被删除，优化知识结构。
Experiments
在ScienceQA和自建跨文档多跳QA数据集上进行评估，比较基础模型、无KG自我对弈和SPARK。指标包括准确率、路径F1和推理深度。训练采用4B参数的Qwen-VL模型，微调3轮，奖励组件消融验证路径忠实度和事实一致性的重要性。结果显示，SPARK在3跳任务中准确率达60.32%，比无KG方法高出近10%。路径F1和推理深度也显著优于对比方法，验证了结构化推理的有效性。
Results
SPARK在ScienceQA达93.0%的准确率，优于无KG方法89.4%。在跨文档多跳QA中，3跳任务准确率提升至60.32%，路径F1也显著提高，验证了推理路径的结构性。奖励组件分析显示，路径忠实度（Rpath）对性能影响最大，模型能有效学习复杂关系。性能随跳数增加下降缓慢，验证了KG结构对多跳推理的支持。边缘修正机制不断优化知识图谱，模型表现持续提升。
Applications
该方法适用于科研自动化、文献综述、智能问答和教育辅助，特别是在跨文档、多模态内容整合场景。未来结合实时知识更新，支持科研辅助和知识管理，推动智能科研工具发展。
Limitations & Outlook
知识图谱构建依赖规则和预训练模型，可能在新颖关系和复杂内容中表现不足。训练成本高，KG修正可能引入噪声。未来需增强自动化鲁棒性和多模态融合效率，扩大应用范围。
Plain Language Accessible to non-experts
想象你在一个大型工厂工作，里面有很多不同的机器和流程。每台机器负责不同任务，比如生产零件、组装或检测。管理者需要知道这些流程之间的关系，比如哪个流程在前，哪个在后，或者哪个流程会影响其他流程。以前，只能看流程图，很多细节都没有标明。现在，工厂用一张“知识图谱”把所有机器、流程和关系都画出来：每个机器和流程是一个点，点之间用线连接，表示它们的关系。这个系统可以自己提出问题，比如“如果这个流程出错，会影响哪些后续步骤？”或者“哪个机器的故障会导致整个生产线停工？”它还会验证答案的正确性，确保每个推理都基于真实关系。通过不断自我练习和修正，这个系统变得越来越聪明，能帮助工厂更高效地运作。这个比喻说明了SPARK如何用结构化的知识图谱帮助机器理解复杂关系，并在科学文献中实现类似的推理。
ELI14 Explained like you're 14
想象你在学校的图书馆里，有很多不同的书、图片和公式。你想知道它们之间的关系，比如哪本书讲的内容和另一本文献有关，或者某个图片是用来说明哪个概念。以前，你只能一页页翻，看不出它们之间的联系。现在，有个聪明的机器人，它用一张“知识图谱”把所有内容连接起来——每个概念、图片、公式都变成一个点，点之间用线连接，表示它们的关系。这个机器人可以自己提出问题，比如“这个公式是用来支持哪个理论的？”或者“这两个研究方法有什么不同？”它还会自己回答问题，确保答案是基于这张关系图的。它不断学习，变得越来越懂得科学文献的内容。通过这种方式，机器人能帮科学家更快理解复杂的研究内容，就像你用一张思维导图整理所有知识一样。这项技术让机器变得更聪明，能像人一样理解科学文章里的关系，推动科学研究变得更高效、更智能。
Abstract
Self-play reinforcement learning has shown strong performance in domains with formally verifiable structure, such as mathematics and coding, where both problem generation and reward computation can be grounded in explicit rules. Extending this paradigm to scientific literature is more challenging: the relationships among multi-modal elements within and across documents are rarely made explicit in text, which makes automatic generation of relational reasoning questions difficult and weakens the reliability of reward signals. We propose SPARK (Self-Play with Asymmetric Reward from Knowledge Graphs), a framework that automatically constructs a unified knowledge graph (KG) from multi-document scientific literature and uses it as the structural basis for self-play. KG paths over multimodal nodes serve as a source for generating relational reasoning questions, and structured facts stored in the KG provide a basis for verifiable reward computation. A single small vision-language model (sVLM) alternates between Proposer and Solver roles under information asymmetry against a fixed KG, a design that we believe can be naturally extended toward online adaptation in future work. We evaluate SPARK on public benchmarks and a self-constructed cross-document multi-hop QA dataset. Results show that SPARK consistently outperforms flat-corpus-based self-play baselines, and the performance gap widens as hop count increases, suggesting that KG-structure grounding contributes to relational multi-hop reasoning beyond what unstructured corpus grounding can provide.