VISTA-Bench: Do Vision-Language Models Really Understand Visualized Text as Well as Pure Text?
VISTA-Bench reveals a significant modality gap in vision-language models when processing visualized text versus pure text, based on multi-task evaluation.
Key Findings
Methodology
Using the systematically designed VISTA-Bench, combining perception, reasoning, and knowledge tasks, the study evaluates over 30 representative models under controlled rendering conditions. The dataset covers hierarchical tasks from basic perception to complex reasoning, ensuring comprehensive diagnosis of model capabilities. Visualized text is rendered with varied fonts and styles, verified via VLM-based filters and manual review, to simulate real-world visual language scenarios. The evaluation compares performance on pure text versus visualized text, analyzing the impact of rendering variations and OCR robustness.
Key Results
- Most models show performance drops exceeding 15% when shifting from pure to visualized text, with some like NEO-9B-SFT dropping over 31%. Conversely, models like MiMo-VL-7B-RL exhibit minimal gaps (~0.3%), indicating robustness. Rendering difficulty, such as font size and style, significantly influences performance, with smaller fonts causing larger declines. Recognition ability correlates strongly with reduced modality gaps, emphasizing the importance of visual text recognition in cross-modal understanding.
- Analysis across tasks reveals that perception remains relatively stable for strong models, but reasoning and knowledge tasks suffer larger declines, often over 10-15 points, highlighting the challenge of complex cross-modal inference from pixel-based language. Variations in rendering parameters amplify the modality gap, demonstrating the sensitivity of models to visual details.
- OCR-based experiments show that high-quality text extraction reduces but does not eliminate the gap, confirming that perceptual robustness—not just OCR accuracy—is critical. Models with weak visual recognition capabilities experience larger performance drops, indicating that visual-text understanding is a bottleneck for current models.
Significance
This research systematically exposes the limitations of current vision-language models in handling visualized text, a common scenario in real-world applications like embedded text in images, documents, and scene understanding. By quantifying the modality gap and identifying core factors such as perceptual robustness and OCR ability, the study provides crucial insights for developing more unified and resilient multi-modal representations. It bridges a critical gap between academic benchmarks and practical needs, guiding future efforts toward models capable of seamless understanding across symbolic and pixel-based language inputs, thereby enhancing applications in OCR, scene text understanding, and multimodal reasoning.
Technical Contribution
The paper introduces VISTA-Bench, a comprehensive, multi-task benchmark with rigorous rendering and validation pipelines, enabling precise measurement of modality gaps. It combines hierarchical tasks across perception, reasoning, and knowledge, with detailed rendering controls and VLM-based fidelity checks. The large-scale evaluation of diverse models reveals the dominant role of perceptual robustness and visual recognition in modality gaps, emphasizing the need for improved visual-text alignment mechanisms. The framework offers a new standard for diagnosing and addressing multi-modal understanding limitations, fostering the development of models with unified language representations.
Novelty
This work is the first to systematically evaluate the impact of visualized text across a broad spectrum of multi-task vision-language understanding, moving beyond isolated OCR or unimodal assessments. The construction of VISTA-Bench, with controlled rendering, hierarchical tasks, and fidelity verification, provides a novel, rigorous framework to quantify the modality gap. It highlights the importance of perceptual robustness and visual recognition, offering new insights into the challenges of unifying symbolic and pixel-based language processing in large models.
Limitations
- The benchmark primarily focuses on static rendered text, lacking dynamic or scene-based scenarios that involve complex visual interactions. The rendering fidelity, while verified, may still not fully capture real-world variability, such as extreme fonts or occlusions.
- Model evaluations are limited to the current set of architectures; future models with different training paradigms or multimodal fusion strategies might behave differently. The evaluation does not yet incorporate real-time or multi-turn interactions.
- Computational costs for rendering and verification are high, and expanding to more diverse visual styles and languages will require additional effort. Further research is needed to generalize findings to more complex, real-world multimodal environments.
Future Work
Future directions include enhancing the visual rendering pipeline to simulate more complex scenes, integrating dynamic and contextual visual language understanding, and developing models with inherently robust visual-text alignment. Extending the benchmark to multilingual and multi-style scenarios will broaden its applicability. Additionally, exploring joint training strategies that fuse symbolic and pixel-based representations could bridge the modality gap, pushing models toward truly unified multi-modal understanding.
AI Executive Summary
The rapid advancement of vision-language models (VLMs) has revolutionized multi-modal understanding, enabling applications from image captioning to visual question answering. However, most evaluations rely on pure-text queries, which do not reflect real-world scenarios where language often appears as visualized text embedded within images. Recognizing this gap, the authors introduce VISTA-Bench, a comprehensive benchmark designed to systematically evaluate models’ ability to understand visualized text across perception, reasoning, and knowledge tasks.
VISTA-Bench employs a carefully curated dataset of 1,500 samples, rendered with diverse fonts and styles, validated through a VLM-based filter and manual review. The benchmark assesses over 30 models, including open-source and proprietary systems, under controlled rendering conditions. Results reveal a pervasive modality gap: models that excel on pure text often suffer performance drops exceeding 15% when processing visualized text. Notably, OCR robustness and visual recognition capabilities are identified as key factors influencing this gap.
Experimental analysis shows that rendering parameters such as font size and style significantly impact model performance, with smaller or stylized fonts causing larger declines. The gap is especially pronounced in reasoning and knowledge tasks, indicating that visualized text understanding involves complex perception and cross-modal inference. OCR-based experiments confirm that perceptual robustness—not just OCR accuracy—is critical for bridging the modality gap.
This work exposes fundamental limitations in current models’ ability to process pixel-based language, emphasizing the need for more resilient visual-text alignment mechanisms. It provides a diagnostic tool and a research pathway toward unified multi-modal representations, with broad implications for real-world applications like scene text understanding, document analysis, and multimodal AI systems. Future efforts will focus on improving rendering fidelity, model robustness, and extending the benchmark to dynamic and multilingual environments, aiming to realize truly integrated visual and linguistic comprehension.
Deep Dive
Abstract
Vision-Language Models (VLMs) have achieved impressive performance in cross-modal understanding across textual and visual inputs, yet existing benchmarks predominantly focus on pure-text queries. In real-world scenarios, language also frequently appears as visualized text embedded in images, raising the question of whether current VLMs handle such input requests comparably. We introduce VISTA-Bench, a systematic benchmark from multimodal perception, reasoning, to unimodal understanding domains. It evaluates visualized text understanding by contrasting pure-text and visualized-text questions under controlled rendering conditions. Extensive evaluation of over 30 representative VLMs reveals a pronounced modality gap: models that perform well on pure-text queries often degrade substantially when equivalent semantic content is presented as visualized text. This gap is further amplified by increased perceptual difficulty, highlighting sensitivity to rendering variations despite unchanged semantics. Overall, VISTA-Bench provides a principled evaluation framework to diagnose this limitation and to guide progress toward more unified language representations across tokenized text and pixels. The source dataset and code are publicly available at https://github.com/QingAnLiu/VISTA-Bench.