GptGet

VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA

TL;DR

VinQA introduces two visual encoding methods, significantly improving visual citation accuracy in long-form multimodal document QA.

cs.CV 🔴 Advanced 2026-06-15 44 views
Young Rok Jang Hyesoo Kong Kyunghwan An Jae Sub Huh Gyeonghun Kim Stanley Jungkyu Choi
AI Reader Arxiv Page Download PDF
multimodal learning document QA visual citation long-form generation dataset construction

Key Findings

Methodology

VinQA constructs a diverse real-world multimodal dataset, employing Page Encoding and Modality Encoding strategies combined with explicit visual citation mechanisms. The proposed M-GroSE evaluation framework assesses answer relevance, completeness, and faithfulness, along with Visual Source F1 for citation accuracy. Fine-tuning Qwen2.5-VL models on VinQA notably boosts performance, with Page Encoding reaching comparable results to Modality Encoding after training, demonstrating the effectiveness of both approaches.

Key Results

  • On the VinQA test set, proprietary models like GPT-4.1 achieved scores of 4.88 (relevance) and 4.27 (completeness). Fine-tuned open-source Qwen2.5-VL models improved to 3.34 and 3.33 on average, narrowing the gap with SOTA.
  • Visual Source F1 scores ranged from 0.55 to 0.72, confirming the accuracy of visual element citations post fine-tuning.
  • Experiments show Modality Encoding performs more robustly on complex long-text, multi-visual-element documents initially, but after training, Page Encoding attains similar performance, indicating their complementary strengths.

Significance

This work advances multimodal document understanding by addressing the challenge of explicit visual element citation within long-form answers. It provides a comprehensive dataset and evaluation framework, fostering progress in automatic document analysis, intelligent QA, and information retrieval industries. The methods enable models to generate more trustworthy and complete responses, crucial for real-world applications.

Technical Contribution

Introduces two innovative visual encoding strategies—Page and Modality Encoding—that effectively incorporate visual regions and layout information into large language models. The integration of multi-dimensional evaluation metrics (M-GroSE) and visual citation accuracy (Visual Source F1) enriches the assessment of multimodal answer quality. Fine-tuning demonstrates adaptability and robustness across complex scenarios, setting a new standard for multimodal long-form QA.

Novelty

First to systematically incorporate explicit visual element citation mechanisms into real-world multimodal document QA, combining Page and Modality encoding strategies. This approach enables models to generate long answers that interleave visual content with supporting text, a significant step beyond prior work limited to short responses or ungrounded answers.

Limitations

  • Models still struggle with extremely complex documents featuring dense visual elements and lengthy texts, especially in precise visual region localization and citation consistency.
  • Training costs are high, particularly for large-scale fine-tuning, limiting widespread deployment.
  • Evaluation metrics, while comprehensive, cannot fully capture semantic alignment and contextual appropriateness of visual citations, requiring further refinement and human validation.

Future Work

Future efforts will focus on developing more efficient visual region detection and citation mechanisms, integrating multimodal pretraining strategies, and expanding datasets across languages and domains. Enhancing evaluation metrics with human-in-the-loop feedback and optimizing models for real-time deployment are also key directions.

AI Executive Summary

In the era of information overload, understanding complex documents with rich visual content is a critical challenge for AI systems. Traditional multimodal large language models (MLLMs) excel at short responses but fall short in effectively utilizing visual elements like charts, images, and diagrams embedded within lengthy texts. This limitation hampers their application in real-world scenarios such as financial analysis, scientific research, and enterprise documentation.

VinQA addresses this gap by proposing two novel visual encoding strategies—Page Encoding and Modality Encoding—that enable models to explicitly cite and incorporate visual elements into long-form answers. These methods are integrated into a comprehensive dataset comprising diverse real-world documents, including academic papers, financial reports, and web pages, each annotated with visual element boundaries and textual descriptions. The dataset facilitates training models to generate detailed, grounded responses that interleave visual content with supporting text.

To evaluate the effectiveness of these approaches, the authors introduce M-GroSE, a multi-dimensional evaluation framework that assesses answer relevance, completeness, and faithfulness, along with Visual Source F1 for citation accuracy. Experimental results demonstrate that fine-tuned models, especially Qwen2.5-VL, significantly outperform baseline models, achieving scores of approximately 3.3–3.6 on the M-GroSE metric after training. Visual citation accuracy also improves markedly, validating the proposed mechanisms.

This work marks a significant step forward in multimodal understanding, enabling models to produce more trustworthy, comprehensive answers in complex document environments. Its implications extend to automated document analysis, intelligent content generation, and enterprise knowledge management. Nonetheless, challenges remain in handling extremely dense visual content and reducing computational costs. Future research will focus on optimizing visual region detection, expanding datasets, and refining evaluation metrics, paving the way for more robust and scalable multimodal AI systems.

Deep Analysis

Background

The evolution of multimodal document question answering (QA) has progressed from early visual question answering (VQA) models like VisualBERT and LXMERT to large multimodal language models (MLLMs) such as GPT-4 and PaLM-E. These models integrate visual and textual data, enabling better understanding of images and text combined. Existing datasets like VQA, DocVQA, and SlideVQA have provided benchmarks, but they mainly focus on short answers or limited visual content, lacking the complexity of real-world documents with long texts and multiple visual elements. Recent advances aim to extend these capabilities, but challenges remain in effectively citing visual content within long responses, especially in diverse domains like finance, research, and web content. VinQA fills this gap by providing a large-scale, multi-domain dataset with detailed visual annotations and a focus on long-form answer generation with explicit visual citations, thus pushing the boundary of multimodal understanding.

Core Problem

Current multimodal QA systems are limited in handling complex, real-world documents that contain lengthy texts and numerous visual elements such as charts, diagrams, and images. These models often generate ungrounded or superficial answers, lacking explicit visual references, which reduces their reliability and interpretability. The core challenge lies in accurately locating, understanding, and citing visual regions within the document context, especially when dealing with diverse layouts and dense visual information. Additionally, existing evaluation metrics do not sufficiently measure the correctness of visual citations, further hindering progress. Addressing these issues requires developing robust encoding strategies, effective visual-region citation mechanisms, and comprehensive evaluation frameworks tailored for long-form, multimodal answers.

Innovation

VinQA introduces two key innovations: (1) Page Encoding, which directly encodes full-page images with visual bounding boxes, preserving layout and spatial information; and (2) Modality Encoding, which extracts text via OCR and crops visual elements for separate encoding, enabling fine-grained understanding. These strategies are coupled with explicit visual-element citation mechanisms, allowing models to reference specific visual regions within answers. Additionally, the paper proposes the M-GroSE evaluation framework, extending previous metrics to assess relevance, completeness, and faithfulness in multimodal answers, along with Visual Source F1 for citation accuracy. This comprehensive approach enables models to generate detailed, visually grounded long responses, a significant step beyond prior short-answer or ungrounded methods.

Methodology

  • �� Construct a large, diverse dataset (VinQA) from real-world documents across multiple domains, annotated with visual element boundaries and descriptions.
  • �� Develop Page Encoding: encode entire pages as images with bounding boxes, preserving layout; input includes page image plus visual region annotations.
  • �� Develop Modality Encoding: apply OCR to extract text, crop visual elements, encode separately; input includes text and cropped images.
  • �� Implement visual element citation: assign unique IDs to visual regions, insert references in generated answers.
  • �� Design the M-GroSE evaluation framework: assess relevance, completeness, faithfulness via gpt-4o, and measure citation accuracy with Visual Source F1.
  • �� Fine-tune Qwen2.5-VL models on VinQA, compare performance across encoding strategies and model types.
  • �� Conduct ablation studies to analyze impact of visual element types, document complexity, and training data size.

Experiments

  • �� Use VinQA’s training set for model fine-tuning, with evaluation on a dedicated test split.
  • �� Compare proprietary models (GPT-4.1, Gemini) and open-source models (Qwen2.5-VL) under both encoding strategies.
  • �� Metrics include M-GroSE scores, Visual Source F1, and human evaluation for answer quality.
  • �� Perform ablation on document complexity, visual element types, and training epochs.
  • �� Analyze the impact of fine-tuning, visual element density, and answer length on performance.
  • �� Validate the models’ ability to cite visual elements accurately and generate comprehensive answers across domains.

Results

  • �� Fine-tuned Qwen2.5-VL models achieved average M-GroSE scores of 3.34 (Page Encoding) and 3.33 (Modality Encoding), showing significant improvement over baseline.
  • �� Visual Source F1 scores increased to 0.55–0.72, confirming enhanced visual citation accuracy.
  • �� Modality Encoding outperformed Page Encoding initially, but after training, both strategies converged in performance, indicating their complementary strengths.
  • �� Proprietary models like GPT-4.1 scored higher overall, but fine-tuning open models narrows this gap, demonstrating adaptability.
  • �� Experiments highlight that models handle complex documents better after training, with performance decreasing slightly for very long contexts, indicating room for further optimization.

Applications

  • �� Enables intelligent document analysis in finance, research, and web content management, automating extraction and interpretation of multimodal data.
  • �� Supports advanced question answering systems that cite visual evidence, improving trustworthiness and user engagement.
  • �� Facilitates automatic report generation, summarization, and content validation in enterprise workflows.
  • �� Long-term, the technology can underpin fully autonomous document understanding systems, transforming industries reliant on complex data interpretation.

Limitations & Outlook

  • �� Despite improvements, models still struggle with extremely dense visual content and long texts, especially in precise localization and citation consistency.
  • �� High computational costs for training and inference limit scalability and real-time deployment.
  • �� Evaluation metrics, while comprehensive, cannot fully capture semantic alignment and contextual appropriateness, necessitating further human-in-the-loop validation.

Plain Language Accessible to non-experts

想象你在看一本非常复杂的说明书，里面不仅有文字，还夹杂着很多图片、图表和示意图。普通的AI助手就像只会读文字的机器人，它不能理解图片中的内容，也不能把图片和文字结合起来帮你解答问题。而VinQA就像一个聪明的朋友，不仅能看懂图片，还能在回答时把图片中的重要信息插进去，就像老师在讲解时会指着图片说话一样。它会把图片和文字变成机器能理解的语言，然后在回答问题时，把相关的图片内容放在合适的位置，让答案更完整、更直观。这就像你问老师“这个步骤怎么做”，老师不仅会用文字讲，还会指着图片一步步解释，让你更容易理解。这样，AI变得更聪明，能帮你解决更复杂的问题，就像一个会看图说话的聪明助手一样。

ELI14 Explained like you're 14

想象你在玩一个超级酷的游戏，里面有很多关卡、任务和隐藏的宝藏。普通的助手只会告诉你“去那里”，但VinQA就像一个聪明的朋友，不仅会告诉你“去那里”，还会指着地图上的宝藏位置，告诉你具体在哪个角落。它能看懂游戏中的图片、地图和任务说明，然后在你问问题时，把图片中的重要信息插进去，帮你更快找到宝藏。它就像一个会看图、会讲故事的伙伴，不仅懂文字，还能理解图片里的秘密。这样，你在游戏中就能更顺利，找到更多隐藏的宝藏，享受更棒的体验。它让AI变得更聪明，就像有个会看图说话的好朋友一样，帮你解决各种难题。

Glossary

Multimodal (多模态)

Refers to processing multiple types of information such as text, images, and audio simultaneously, enabling richer understanding. In this paper, it combines textual and visual data for document QA.

Describes the input data types handled by the models.

Visual Citation (视觉引证)

A mechanism where specific visual elements are explicitly referenced within generated answers to ensure clarity and grounding. It improves answer trustworthiness.

Key to enabling models to cite images, charts, or diagrams in responses.

Page Encoding (页面编码)

A strategy that encodes entire document pages as images with annotated visual regions, preserving layout and spatial information for model input.

One of the two main encoding methods proposed.

Modality Encoding (模态编码)

A method that extracts text via OCR and crops visual elements, encoding them separately to facilitate fine-grained understanding.

The second main encoding strategy.

M-GroSE (Multi-modal Grounded QA Scoring Evaluator)

An evaluation framework that assesses answer relevance, completeness, faithfulness, and visual citation accuracy in multimodal QA.

Used to systematically evaluate model outputs.

Open Questions Unanswered questions from this research

  • 1 如何进一步提升模型在极端复杂场景（如大量视觉元素和长文本）中的理解能力仍是未解难题，特别是在视觉区域定位和引证一致性方面。
  • 2 现有评估指标难以完全衡量视觉引证的语义一致性和上下文适应性，未来需要结合人类评审进行优化。

Applications

Immediate Applications

智能文档分析

自动解读财务报告、科研论文等多模态文档，提升信息提取效率和准确性。

企业知识管理

构建智能问答系统，帮助员工快速获取多模态资料中的关键信息。

Long-term Vision

智能内容生成

实现自动化生成多模态长文本内容，应用于新闻、报告、教育等行业。

Abstract

Real-world documents combine text with tables, charts, photographs, and diagrams arranged in diverse layouts, yet existing research on multimodal large language models (MLLMs) for document QA predominantly produces text-only responses, underutilizing these visual elements. We introduce VinQA, a dataset for long-form answer generation where cited visual elements are explicitly interleaved with their supporting text and grounded in relevant document pages. To support this task, we study two encoding methods for feeding raw document page images into an MLLM, along with their visual-element citation mechanisms: (1) Page Encoding, which directly encodes full-page images with bounding boxes of visual elements and treats these boxed regions as citable units; and (2) Modality Encoding, which parses each page to extract text and crop visual elements, encodes them separately, and uses these cropped elements as citable units. In our experiments, we propose M-GroSE, a multimodal evaluation framework extending GroUSE to assess answers along four dimensions: completeness, answer relevancy, faithfulness, and unanswerability. We additionally report Visual Source F1 to directly measure visual citation accuracy. Although proprietary frontier models still achieve the best overall scores on the VinQA test split, fine-tuning open Qwen2.5-VL models on the training split substantially improves their performance and narrows this gap. Modality Encoding is initially more robust for complex documents with long text, many visual elements, and diverse citation requirements. After training on VinQA, however, Page Encoding reaches a comparable level, competing effectively even without the explicit parsing used in Modality Encoding. Finally, Visual G-Eval, an MLLM-based judge, confirms that fine-tuned models insert visual elements at semantically appropriate positions with faithful supporting text.

cs.CV cs.AI