ChartWalker: Benchmarking the Cross-Chart RAG Task with Hierarchical Knowledge Graphs
ChartWalker constructs hierarchical knowledge graphs and uses structure-aware sampling to improve cross-chart multi-hop reasoning, boosting multimodal QA.
Key Findings
Methodology
This paper introduces ChartWalker, which builds hierarchical knowledge graphs by extracting entities and relations from charts using pretrained vision-language models. Entities are organized into layers based on granularity, then merged into a global graph. A structure-aware path sampling algorithm ensures semantic coherence and controllable granularity along reasoning paths. These paths serve as supervision for generating multi-hop question-answer pairs, combining explicit reasoning chains with multimodal evidence. The framework effectively reduces semantic drift and enhances logical consistency in complex reasoning scenarios.
Key Results
- On the ChartWalker-Bench, the best model achieves only 64% accuracy, highlighting task difficulty. Multi-hop reasoning accuracy drops below 30% in complex queries, confirming the challenge of maintaining semantic coherence. The path sampling strategy significantly outperforms random walk baselines in generating meaningful reasoning paths. The average reasoning steps per question are 2.81, with 2.30 source charts, reflecting realistic complex scenarios.
- Across different RAG models, performance gaps are evident, with the best models struggling on multi-hop, multi-granularity tasks. Ablation studies show that hierarchical graph construction and structure-aware sampling are critical for improving reasoning quality. The proposed approach demonstrates robustness in handling diverse query types and reasoning depths.
- The ChartWalker-Agent, a dynamic retrieval strategy based on visual language models, outperforms static retrieval methods, especially on complex reasoning tasks, indicating the importance of adaptive evidence collection for multimodal reasoning.
Significance
This work addresses a critical gap in multimodal reasoning evaluation by providing a challenging, structured benchmark for cross-chart QA. The hierarchical knowledge graph approach and structure-aware sampling enable more reliable and interpretable reasoning paths, facilitating progress in AI systems capable of understanding and integrating complex visual and textual data. The benchmark supports systematic assessment of retrieval and reasoning modules, pushing the frontier of multimodal AI in scientific, business, and policy domains. It also offers insights into the design of more robust, explainable AI systems that can handle real-world, multi-source data analysis tasks.
Technical Contribution
The paper introduces a novel hierarchical graph construction method tailored for chart data, capturing multi-granular entities and relations. The structure-aware sampling algorithm enforces semantic coherence, reducing path drift and improving reasoning reliability. The integration of these components into a comprehensive framework advances the state-of-the-art in multi-hop multimodal reasoning and QA generation. Additionally, the creation of the ChartWalker-Bench dataset provides a valuable resource for benchmarking future models, fostering research in structured multimodal reasoning.
Novelty
This is the first work to explicitly incorporate hierarchical, multi-granular knowledge graphs for cross-chart reasoning, addressing the limitations of previous semantic similarity-based path generation. The structure-aware sampling algorithm ensures paths are both semantically meaningful and controllable in granularity, representing a significant step forward in generating reliable reasoning chains for multimodal QA. Unlike prior benchmarks, which focus on simple or unstructured data, this approach emphasizes explicit reasoning structures aligned with real-world complexity.
Limitations
- The entity extraction quality heavily depends on pretrained vision-language models, which may misidentify or omit entities, affecting path quality. In highly dense or complex charts, entity resolution remains challenging.
- Path coherence is constrained by current sampling strategies, which may still produce paths with subtle semantic drift in some cases, especially with ambiguous or noisy data.
- Computational costs of hierarchical graph construction and path sampling are high, limiting scalability for extremely large datasets or real-time applications.
Future Work
Future efforts will focus on end-to-end training of the entity extraction and path sampling modules, incorporating reinforcement learning to optimize path quality. Exploring dynamic knowledge updates and real-time reasoning capabilities will enhance adaptability. Extending the benchmark to include more diverse domains and multimodal data types, such as videos or interactive charts, will further broaden its applicability. Additionally, integrating explainability modules to interpret reasoning paths will improve system transparency.
AI Executive Summary
In the era of data-driven decision making, charts serve as vital tools for visualizing complex information across scientific, business, and political fields. However, leveraging multiple charts for comprehensive analysis remains a significant challenge for AI systems. Existing question-answering models often rely on structured tables or simple key-point extraction, which fail to capture the rich visual and relational complexity inherent in charts. These limitations hinder their ability to perform reliable, multi-step reasoning across diverse visual data sources.
To address this, the authors introduce ChartWalker, a novel framework that constructs hierarchical knowledge graphs from chart data. By extracting entities and relations using pretrained vision-language models, ChartWalker organizes information into layered structures that reflect different levels of granularity. This organization enables the generation of semantically coherent multi-hop reasoning paths through a structure-aware sampling algorithm. These paths serve as explicit reasoning chains, guiding the generation of high-quality, multi-modal question-answer pairs that are both logically consistent and visually grounded.
Building upon this methodology, the authors develop ChartWalker-Bench, a comprehensive benchmark comprising 564 multi-hop QA instances across various domains and query types. Extensive evaluations reveal that current models achieve only up to 64% accuracy, with performance dropping sharply on complex reasoning tasks. This underscores the difficulty of the task and the necessity for more sophisticated reasoning strategies. To facilitate future research, the authors also propose ChartWalker-Agent, a dynamic retrieval system leveraging visual language models that adaptively explore evidence sources, outperforming static retrieval approaches.
Overall, this work advances the field of multimodal reasoning by introducing a structured, interpretable approach to cross-chart analysis. It provides a challenging benchmark and effective tools for developing more robust, explainable AI systems capable of handling real-world, multi-source visual data. Despite promising results, challenges remain in entity extraction accuracy, path coherence, and computational efficiency, guiding future directions in this rapidly evolving area.
Deep Analysis
Background
图表作为一种直观表达定量信息的重要工具，广泛应用于科学研究、商业分析、新闻报道和政策制定中。早期研究主要集中在结构化表格的问答（如Pasupat & Liang, 2015）和简单图表理解（Li et al., 2024），利用符号推理或文本匹配解决基础问题。随着视觉语言模型（VLM）和多模态技术的发展，复杂图表推理成为可能（Masry et al., 2025），但仍面临多源信息整合、路径生成和逻辑连贯性等挑战。现有基准（Yang et al., 2025）多依赖语义相似度，路径生成缺乏结构感知，导致推理路径不稳定，难以全面评估模型能力。这些限制阻碍了跨图表复杂推理的实际应用。
Core Problem
跨图表推理的核心难题在于如何构建具有逻辑连贯性和粒度控制的推理路径。传统方法多采用随机游走或语义匹配，容易偏离主题或生成无效路径，限制了多跳推理的准确性和可解释性。图表的多模态特性和信息密度进一步增加了路径采样的复杂度，如何确保路径的语义连贯和信息层次的合理性成为亟待解决的问题。这些技术瓶颈限制了多源、多跳推理系统在实际场景中的推广。
Innovation
本文的主要创新包括：• 构建层级化知识图谱，将图表中的实体按粒度组织，保持信息的层次结构，支持多层次推理；• 设计结构感知采样算法，确保路径在语义和粒度上的连续性，有效避免路径漂移；• 基于路径的多模态问答生成机制，提升问答的逻辑性和多跳推理能力。这些创新结合了预训练模型和知识图谱技术，为多模态推理提供了新思路，显著优于传统随机游走和语义匹配方法。
Methodology
- �� 利用预训练视觉语言模型（如VinVL）提取图表实体及其粒度层级信息；• 构建局部图表知识图谱，连接不同粒度的实体，反映图表结构关系；• 将局部图谱融合成全局层级知识图谱，支持跨图表推理；• 设计路径采样算法，基于改良PageRank（考虑实体连接度和源多样性）选择路径起点，沿路径保持语义连贯，控制粒度变化；• 采样多条路径，结合路径信息生成问答对，确保逻辑合理、信息丰富；• 通过自动质量筛选，过滤高质量问答，最终构建ChartWalker-Bench。
Experiments
在ChartWalker-Bench上，采用多种RAG模型（如视觉语言模型和Transformer）进行评估，指标包括正确率、推理步数和路径连贯性。模型在复杂推理任务中的表现验证了路径采样策略的有效性。对比随机游走，结构感知路径显著提升路径的语义连贯性。消融实验表明，层级组织和路径采样是性能提升的关键因素。实验还分析了不同推理难度场景下模型的表现差异，确保结果的全面性和可靠性。
Results
最优模型在基准上的最高正确率为64%，多跳推理在复杂场景中准确率低于30%，显示出任务的挑战性。路径采样策略明显优于随机游走，提升路径的语义连贯性。模型在复杂推理中的表现验证了路径结构设计的有效性，表明多源、多跳推理潜力巨大。路径粒度控制增强了问答的可解释性，为未来模型优化提供了方向。
Applications
该方法适用于科学研究、市场分析、政策评估等多领域场景，帮助用户从多个图表中快速提取关联信息，支持复杂推理和决策。系统可集成于智能问答平台，实现多源信息的自动检索和整合，提升决策效率。未来还可结合自动图表生成和知识动态更新，推动智能数据分析的普及和应用。
Limitations & Outlook
当前路径采样依赖实体提取的准确性，模型在极端复杂或信息密集的图表中仍面临路径偏离和语义漂移问题。路径长度受限，超长推理链可能导致信息遗失。未来需优化路径筛选机制，提升模型的泛化能力和推理效率。
Plain Language Accessible to non-experts
想象你在厨房里准备一道复杂的菜肴，厨房里有许多不同的食材和工具。每个食材代表不同的信息，比如水果、蔬菜、调料，而工具像是图表中的关系。你需要根据菜谱（问题）找到正确的食材和步骤，同时确保每一步都合理，比如先切水果再炒菜。为了做出好菜，你会按照一定的顺序和逻辑，逐步选择食材和操作。ChartWalker就像一个聪明的厨师助手，它能帮你把所有食材和工具的关系整理成一个层次结构（知识图谱），然后根据菜谱中的提示，找到最合适的步骤（路径）来完成菜肴。这样，无论菜谱多复杂，它都能帮你理清思路，确保每个步骤都合理，最后做出美味佳肴。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，拼图上有很多不同的图片和线索。你的任务是找到正确的拼图块，把它们拼在一起，形成完整的画面。但是，有时候线索会很混乱，拼图块也很多，怎么才能找到正确的拼法呢？这时，你需要一个聪明的朋友（ChartWalker），它会帮你把所有线索按照层次整理，比如大图、小图、细节，然后帮你找到一条条合理的拼图路径。每条路径就像是拼图的线索链，告诉你下一步该拼哪个块，确保拼出来的画面是连贯的。这个朋友还能帮你避免走弯路，不会拼错。通过这种方式，你就能更快、更准确地完成拼图，拼出漂亮的画面。
Glossary
Knowledge Graph (知识图谱)
一种用节点和边表示实体及其关系的图结构，用于组织和推理信息。
本文中用于构建图表实体的层级关系和推理路径。
Hierarchical Knowledge Graph (层级知识图谱)
将实体按粒度层次组织的知识图谱，便于多层次推理。
用于组织图表中的多粒度实体信息。
Structure-aware Sampling (结构感知采样)
考虑路径语义连续性和粒度变化的路径采样算法。
确保生成的推理路径语义连贯。
Retrieval-Augmented Generation (检索增强生成)
结合检索机制与生成模型，提高问答的准确性和逻辑性。
本文中的核心问答生成框架。
Multi-hop Reasoning (多跳推理)
跨越多个实体或关系进行连续推理的过程。
实现复杂跨图表推理任务的关键技术。
Open Questions Unanswered questions from this research
- 1 如何进一步提升路径采样在极端复杂图表中的语义连贯性仍是挑战，尤其是在信息密度极高或结构复杂的场景下，模型的推理路径可能偏离主题，未来需引入更智能的路径筛选和优化机制。
Applications
Immediate Applications
科学数据分析
帮助研究人员从多个图表中快速提取关联信息，支持复杂的科学推理和决策。
市场调研
辅助分析师整合不同市场报告中的图表数据，生成有逻辑的结论，提升报告效率。
Abstract
Cross-Chart Retrieval-Augmented Generation (RAG) is critical for complex multi-modal analytical tasks in scientific, business, and political domains. However, existing benchmarks either focus on tables, which are well-structured and textualized, or generate cross-chart questions by simply extracting key points, which often induces lexical overlap between queries and evidence and yields logically inconsistent reasoning chains. To address this, we introduce ChartWalker, a novel framework for constructing challenging cross-chart RAG tasks. ChartWalker features a hierarchical knowledge graph construction method tailored to charts, which organizes entities and relations by granularity to preserve analytical structure. We then propose a structure-aware sampling algorithm that synthesizes semantically coherent, multi-hop reasoning paths, enabling explicit control over query difficulty and granularity for QA generation. Built with this framework, we release ChartWalker-Bench, a comprehensive benchmark spanning diverse domains and cross-chart query types. Extensive evaluations across major RAG paradigms reveal significant performance gaps, underscoring the benchmark's difficulty and utility. Furthermore, we provide ChartWalker-Agent, an agentic baseline to facilitate analysis and inspire future system design.