ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for Complicated Chart Reasoning
Proposed ChartX benchmark and ChartVLM model achieve near GPT-4V performance in complex chart reasoning.
Key Findings
Methodology
This work constructs ChartX, a comprehensive multi-modal dataset with 48K high-quality charts covering 18 types, 7 tasks, and 22 disciplines, integrating image, CSV, Python code, and text. The ChartVLM model employs a dynamic instruction adapter to select tasks, leveraging a Pix2Struct-based encoder-decoder architecture fine-tuned with LoRA on Vicuna and Qwen models. Evaluation metrics include EM, GPT-acc, and GPT-score, assessing perception and reasoning capabilities across tasks such as structural extraction, QA, description, summarization, and redrawing.
Key Results
- ChartVLM surpasses mainstream multi-modal models, achieving 40.71% accuracy in structure extraction and 94.27% in QA, approaching GPT-4V levels in several tasks.
- In multi-type, multi-task settings, the model demonstrates superior performance, validating the effectiveness of structure-guided reasoning.
- Dynamic task scheduling and structure integration improve interpretability and efficiency, showing strong generalization across disciplines.
Significance
This research addresses the gap in multi-modal models' ability to perform complex, interpretable chart reasoning, providing a systematic evaluation framework and a high-performance architecture. It advances AI's capacity for scientific data analysis, business intelligence, and decision support, emphasizing explainability and multi-task adaptability, crucial for real-world deployment.
Technical Contribution
The paper introduces ChartX as a high-quality, multi-disciplinary benchmark, and develops ChartVLM, which integrates a structure-aware encoder-decoder with a flexible instruction adapter. This design reduces external module reliance, enhances interpretability, and supports multi-task end-to-end reasoning, setting new standards in chart understanding.
Novelty
First to create a comprehensive, multi-disciplinary chart evaluation dataset and propose a structure-guided, interpretable multi-task model. The approach emphasizes internal structure extraction for reasoning, contrasting with prior methods relying heavily on external modules or limited datasets, marking a significant innovation.
Limitations
- The model's performance on highly complex or multi-dimensional charts remains limited due to data and architecture constraints. High computational costs hinder real-time deployment, and support for dynamic or interactive charts is currently insufficient. Future work should focus on expanding data diversity, optimizing efficiency, and enhancing multi-modal interaction capabilities.
Future Work
Future directions include expanding dataset diversity, integrating reinforcement learning for adaptive reasoning, and improving multi-modal fusion for dynamic, interactive charts. Efforts will also target reducing computational costs and deploying models in real-world applications, such as scientific research and enterprise analytics.
AI Executive Summary
In recent years, multi-modal large language models (MLLMs) have demonstrated impressive capabilities across vision-language tasks. However, their proficiency in complex chart reasoning—extracting detailed numerical and structural information from visual data and engaging in logical inference—remains limited. Existing benchmarks like ChartQA and ChartBench offer partial insights but lack comprehensive coverage of diverse chart types and tasks, especially in interpretability and multi-disciplinarity.
To address this gap, the authors introduce ChartX, a large-scale, multi-task evaluation dataset encompassing 18 chart types, 7 tasks, and 22 disciplines, with 48K multi-modal samples. The dataset integrates image, CSV, Python code, and text modalities, enabling detailed assessment of models' perception and reasoning abilities. Building on this, the authors propose ChartVLM, a novel multi-modal model that employs an instruction adapter to dynamically select tasks, combined with a Pix2Struct-based encoder-decoder architecture fine-tuned via LoRA on Vicuna and Qwen. This design allows end-to-end, interpretable, multi-task reasoning, emphasizing internal structure extraction to improve transparency.
Experimental results demonstrate that ChartVLM significantly outperforms existing multi-modal models, achieving structure extraction accuracy of 40.71% and question-answering accuracy of 94.27%. Its performance approaches that of GPT-4V in several tasks, highlighting the effectiveness of structure-guided reasoning. The model's flexible architecture facilitates multi-task adaptation, making it suitable for scientific, industrial, and commercial applications where interpretability and accuracy are critical.
This work advances the state-of-the-art in chart understanding, providing a robust evaluation framework and a highly capable model. Future efforts will focus on expanding data diversity, optimizing computational efficiency, and extending support for dynamic and interactive charts, aiming to realize AI systems capable of complex, explainable data reasoning in real-world scenarios.
Deep Analysis
Background
随着多模态大模型（MLLMs）在视觉-语言任务中的快速发展，图表理解作为数据可视化的重要应用逐渐成为研究热点。早期工作如Deplot、StructChart等主要集中在图表到表格或文本的转换，强调信息提取能力，但缺乏系统化的推理评估。近年来，Pix2Struct等模型引入编码器-解码器架构，提升了结构化信息的提取能力，但在复杂推理和多任务场景中仍有限。现有评测集如ChartQA、ChartBench等覆盖范围有限，难以全面衡量模型的多样性和推理深度。本文旨在弥补这一空白，构建更丰富的评估体系，推动图表理解的深度发展。
Core Problem
当前多模态模型在图表理解中存在结构信息提取不足、推理能力有限、解释性差等问题。尤其在复杂图表、多任务场景下，模型难以同时兼顾信息的准确性和推理的透明度。缺乏统一的评估标准限制了模型的比较和优化，导致实际应用中效果不理想。解决这些瓶颈对于数据驱动决策、科学研究和商业分析具有重要意义。
Innovation
本研究提出了ChartX评估集，涵盖多类型、多任务、多学科，丰富了评估场景。引入结构信息引导推理的ChartVLM模型，通过指令适配器实现任务动态调度，结合Pix2Struct架构支持多任务端到端推理，显著提升结构提取和推理的准确性。模型内部结构透明，增强了可解释性，突破了传统依赖外部模块的局限。此创新框架为多模态图表理解提供了新思路，兼顾性能与解释。
Methodology
- �� 构建ChartX数据集，采集48K多模态图表，涵盖多学科、多类型，融合图像、CSV、Python、文本。
- �� 设计结构化信息提取模块（Pix2Struct基础架构），实现图表到CSV的端到端转换。
- �� 开发指令适配器，利用GPT-3.5生成多样任务指令，结合微调模型支持多任务调度。
- �� 采用LoRA微调Vicuna和Qwen模型，增强模型在问答、描述、总结等任务中的表现。
- �� 设计多任务评估指标，包括EM、GPT-acc、GPT-score，全面衡量模型性能。
- �� 进行大量实验，验证模型在结构提取、复杂推理、多任务适应性上的优越性。
Experiments
使用ChartX评估集，比较ChartVLM与多模态大模型（如GPT-4V、LLaVA）在结构提取、问答、描述、总结、重绘等任务上的表现。采用指标如EM、GPT-acc、GPT-score，进行多任务、多类型图表的性能评估。实验中调优指令提示，验证模型的泛化能力和多任务适应性。通过消融实验分析指令适配器和结构引导机制的贡献，确保模型在复杂场景中的鲁棒性。
Results
ChartVLM在结构提取任务中达40.71%的准确率，问答任务中达94.27%的正确率，显著优于现有模型。模型在多类型、多任务场景中表现优异，部分指标与GPT-4V持平或接近，验证了结构信息引导推理的有效性。模型在复杂推理和多模态融合方面展现出优越的性能，证明了其在科学和商业应用中的潜力。
Applications
模型可应用于科学数据分析、金融市场预测、商业智能等领域，帮助用户自动理解复杂图表、提取关键信息、生成报告。只需提供图表和指令，即可实现自动推理和可视化重绘，极大提升工作效率。未来还可结合交互式界面，支持动态数据探索和多模态交互，推动智能决策。
Limitations & Outlook
模型在极端复杂或多维图表中的表现仍有限，受限于训练数据的多样性和模型容量。高算力需求限制了模型在实际场景中的快速部署，需优化模型结构和推理效率。对某些特殊图表类型（如3D、动态交互式图表）支持不足，未来应考虑多模态融合与交互能力提升。
Plain Language Accessible to non-experts
想象你在厨房做饭，所有食材和工具都摆在桌子上。你需要先认出每个食材（比如蔬菜、肉类），然后根据菜谱（任务指令）决定怎么用它们。模型就像一个聪明的厨师，能快速识别食材（结构信息提取），理解菜谱（推理任务），最后做出一道美味的菜（生成答案或图表）。这个过程强调理解每个步骤的原因和关系，而不是盲目操作。通过训练，厨师学会了多种菜谱，能应对不同的菜肴需求，甚至能自己创新新菜。
ELI14 Explained like you're 14
想象你在学校的科学课上，老师给你一张复杂的图表，问你一些问题，比如哪个时间点销量最高，或者这个图表讲的是什么故事。你需要先看懂图表上的信息，比如数值、标题和类型，然后用自己的话回答问题。这就像你用放大镜仔细观察，然后用笔写出答案。这个模型也是一样，它先把图表的结构信息找出来（像用放大镜看），然后再根据问题进行推理，得出答案。它还可以画出类似的图表，帮你更好理解数据。这个技术让电脑变得像一个聪明的助手，能帮你快速理解各种复杂的图表，特别是在科学和商业中非常有用。
Glossary
Multi-modal Large Language Model (多模态大语言模型)
一种结合图像、文本等多种信息输入，进行自然语言理解和生成的深度学习模型。
本文中的ChartVLM即为此类模型的实例。
结构化信息 (Structured Data)
以明确格式（如CSV）组织的数据信息，便于模型理解和推理。
模型通过结构化信息实现图表到文本的端到端转换。
指令适配器 (Instruction Adapter)
一个模块，用于根据用户指令动态选择模型任务，提升模型的灵活性和可解释性。
在ChartVLM中实现多任务调度。
Pix2Struct
一种基于编码器-解码器架构的图像到结构化信息转换模型。
作为ChartVLM的基础架构之一。
LoRA (Low-Rank Adaptation)
一种微调技术，用于在预训练模型基础上快速适应新任务。
用于微调Vicuna和Qwen模型。
Open Questions Unanswered questions from this research
- 1 如何进一步提升模型在动态交互式图表中的表现仍未解决，尤其是在多模态融合和实时交互方面。未来需要探索更高效的结构化信息编码与推理机制，以应对更复杂的图表类型和场景。
Applications
Immediate Applications
科学数据分析
帮助科研人员自动理解复杂实验图表，提取关键数值，生成报告，提升科研效率。
商业智能
自动解读财务报表、市场趋势图，辅助决策，减少人工分析时间。
Long-term Vision
智能决策助手
未来模型能在实时场景中理解多模态数据，提供深度分析和建议，成为企业和科研的智能助手。
Abstract
Recently, many versatile Multi-modal Large Language Models (MLLMs) have emerged continuously. However, their capacity to query information depicted in visual charts and engage in reasoning based on the queried contents remains under-explored. In this paper, to comprehensively and rigorously benchmark the ability of the off-the-shelf MLLMs in the chart domain, we construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data. Besides, we develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns, such as reasoning tasks in the field of charts or geometric images. We evaluate the chart-related ability of mainstream MLLMs and our ChartVLM on the proposed ChartX evaluation set. Extensive experiments demonstrate that ChartVLM surpasses both versatile and chart-related large models, achieving results comparable to GPT-4V. We believe that our study can pave the way for further exploration in creating a more comprehensive chart evaluation set and developing more interpretable multi-modal models. Both ChartX and ChartVLM are available at: https://github.com/Alpha-Innovator/ChartVLM