SynthDocBench: Controlled Benchmark for Long-Context Visual Document Understanding
SynthDocBench uses synthetic long documents with controlled factors to reveal three key failure modes of vision-language models in long-context understanding.
Key Findings
Methodology
SynthDocBench employs large language models (LLMs) to generate six layout archetypes of synthetic long documents, systematically varying length, layout complexity, modality, and question types as independent axes. A combinatorial design ensures each factor's influence is isolated. Documents contain multi-modal content, including charts and text, with dual-layer structure—rendered visualizations and structured metadata—guaranteeing deterministic answers. Evaluation involves seven cutting-edge VLMs (e.g., Gemini-3.1-Pro, GPT-5.4), with a 40% random override to prevent spurious correlations. Results expose three failure modes: performance degradation with increasing document length, positional sensitivity (middle third hardest), and collapse of chart comprehension in long documents, indicating models overfit benchmark artifacts rather than achieving robust long-context understanding.
Key Results
- Models show sharp performance drops—up to 8.3 percentage points—as document length increases. Five out of six models find the middle third of documents hardest, with a negative Early→Late trend (up to 8.3 points decline). Chart comprehension collapses in long documents, with a 46-point gap compared to OCR-based methods. Performance across difficulty levels (L1-L5) declines monotonically, especially in high-difficulty questions. Multimodal questions, such as cross-modal reasoning, remain particularly challenging, exposing core weaknesses in visual perception and evidence retrieval.
- Performance drops significantly at higher difficulty levels, with an average decline of over 20 percentage points at L5. The models struggle with precise chart reading and cross-page evidence integration, especially in complex reasoning tasks. The positional bias analysis shows the middle sections are consistently hardest, with performance dropping 5-18 points, indicating a systemic challenge in long-range reasoning and evidence localization.
- Visual perception remains the primary bottleneck: models perform 46 percentage points worse on chart questions when relying solely on vision compared to OCR, confirming that pixel-level understanding is critical. These results highlight the importance of improving visual encoding and reasoning capabilities for long, multimodal documents.
Significance
This work introduces a highly controllable synthetic benchmark that enables detailed diagnosis of vision-language models’ failures in long-context, multimodal understanding. Unlike prior benchmarks limited to single-page or real documents, SynthDocBench systematically manipulates key factors, revealing fundamental weaknesses such as positional sensitivity and collapse of chart comprehension. The findings challenge the assumption that current models are robust, emphasizing the need for new architectures and training strategies to handle complex, multi-page visual documents. This has broad implications for deploying AI in real-world scenarios like legal, financial, and scientific document analysis, where understanding long, multi-modal content is crucial. The benchmark sets a new standard for diagnostic evaluation, fostering more resilient and interpretable models.
Technical Contribution
The core technical innovation lies in the integrated synthesis pipeline that combines large language models with structured metadata and visualization rendering. This enables end-to-end generation of diverse, high-fidelity long documents with precise ground-truth answers. The dual-layer document design ensures deterministic QA derivation, facilitating rigorous evaluation. The combinatorial control over length, layout, modality, and question type allows systematic ablation studies. Evaluation with GPT-5 as a judge, along with multiple models, provides robust performance insights. The approach advances beyond prior real-data benchmarks by enabling controlled, scalable, and interpretable diagnostics of long-context multimodal understanding, setting a foundation for future research.
Novelty
This is the first work to systematically generate and evaluate synthetic long documents with controllable factors, integrating multi-layout archetypes, multimodal content, and structured metadata. Unlike existing benchmarks that rely solely on real data, SynthDocBench offers a diagnostic platform that isolates specific failure modes, such as positional sensitivity and chart comprehension collapse, which are difficult to detect otherwise. Its combination of automated generation, deterministic answers, and multi-factor control represents a significant methodological leap in long-text multimodal evaluation.
Limitations
- While synthetic documents provide controlled variation, they may not fully capture the complexity and diversity of real-world documents, potentially limiting ecological validity.
- Model performance on synthetic data might not directly translate to real scenarios due to generation biases or oversimplification of certain layout features.
- Current evaluation focuses on vision-language models; multi-modal fusion robustness and generalization to noisy real data remain open challenges.
Future Work
Future directions include integrating real-world long documents to validate findings, expanding the diversity of layout archetypes, and developing models explicitly optimized for long-range multimodal reasoning. Additionally, exploring adaptive training strategies and interpretability techniques will be crucial to enhance model robustness in practical applications.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家非常复杂的工厂里工作，这个工厂有很多不同的房间（布局），每个房间里有不同的机器（图表、文本），而你需要找到某个特定的零件（答案）。以前，我们用一些简单的工具（模型）在单个房间里找东西，但当工厂变得更大、更复杂时，这些工具就变得不够用了。现在，科学家们用一种叫SynthDocBench的“智能工厂”模拟系统，能生成各种复杂的房间和机器，让工具测试在不同情况下的表现。通过这种方法，他们发现工具在工厂的中间区域最难找到东西，或者在长长的走廊里理解图表变得很困难。这帮助他们知道，未来需要改进工具，让它们在复杂环境中也能表现得更好，就像让工厂的机器人变得更聪明一样。
ELI14 Explained like you're 14
想象你在一个超级复杂的学校里，有很多不同的教室（布局），每个教室里有不同的老师和学生（文本和图表）。你要找到某个特定的答案，比如哪个学生得了最高分。以前，老师们用简单的方法在一个教室里找答案，但当学校变得更大、更复杂时，这些方法就不够用了。科学家们发明了一种叫SynthDocBench的“模拟学校”，它可以创造各种不同的教室和场景，让老师们测试他们的搜索技巧。结果发现，学生们在学校的中间区域最难找到答案，而且理解长长的报告和复杂的图表也很困难。这告诉我们，要让老师们更聪明，就得教他们如何在复杂的环境中快速找到信息，变得更厉害！
Abstract
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.