LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
LogicOCR evaluates large multimodal models' logical reasoning on text-rich images using automated image generation and cropping, achieving 73% success rate.
Key Findings
Methodology
This study constructs the LogicOCR benchmark with 2780 questions, divided into LogicOCR-Gen (1100 multiple-choice on generated images) and LogicOCR-Real (1680 free-form on real images). Using a custom pipeline, Qwen2.5-14B and GPT-Image-1 generate diverse, realistic images with varied layouts, fonts, and backgrounds, ensuring contextual relevance. Manual verification ensures quality. Multiple models, including Qwen2.5-VL, InternVL, and Gemini-2.5-Pro, are evaluated under Chain-of-Thought (CoT) and direct-answer settings. Analysis shows models still lag in visual-text comprehension, especially under visual-text orientation perturbations. The proposed TextCue method leverages attention maps and a text segmentation model to automatically crop key regions, improving reasoning performance.
Key Results
- Models achieve an average accuracy of XX% on LogicOCR-Gen, significantly below pure text reasoning benchmarks. CoT improves performance marginally, indicating flawed reasoning paths. Test-time scaling boosts accuracy by up to 3%. Under rotation or occlusion, accuracy drops to near-random, highlighting OCR robustness issues. TextCue enhances LLaVA-OV-1.5-8B accuracy by 1.8%, validating its effectiveness.
- In LogicOCR-Real, model accuracy drops further, with significant challenges in multi-hop and complex reasoning. Performance varies across reasoning types—numerical, temporal, decision—highlighting the difficulty of true multimodal reasoning. The analysis emphasizes the persistent gap between current models and human-level understanding in complex visual-text scenarios.
- Multi-dimensional analysis reveals sensitivity to visual-text orientation and limited multi-modal fusion. TextCue effectively directs models' attention to critical regions, offering a promising direction for future improvements. Overall, the study underscores the need for better visual-text integration and robust OCR in multimodal reasoning systems.
Significance
This work addresses a critical gap in evaluating multimodal models' reasoning in text-dense images, proposing a large-scale, diverse benchmark. By combining automated image synthesis with cropping-based enhancement, it provides a systematic way to measure and improve models’ capabilities. The findings reveal that despite progress, models still struggle with complex visual-text reasoning, especially under real-world conditions. The benchmark and methods serve as valuable tools for academia and industry, guiding future research toward more intelligent, reliable multimodal AI systems capable of real-world document understanding, automated question answering, and scene comprehension.
Technical Contribution
The paper introduces an automated pipeline combining Qwen2.5-14B and GPT-Image-1 for generating diverse, realistic text-rich images. It proposes TextCue, a training-free cropping method leveraging attention maps and a text segmentation model, to automatically locate and enlarge key regions, enhancing model perception without additional training. The benchmark covers multiple reasoning categories and scenarios, providing a comprehensive evaluation platform. Extensive experiments demonstrate the effectiveness of the cropping approach, with significant accuracy improvements across models. This work advances multimodal reasoning by integrating automated data synthesis, attention-based localization, and robust evaluation, opening new avenues for scalable, real-world AI applications.
Novelty
This is the first comprehensive benchmark assessing large multimodal models' logical reasoning on complex, text-rich images with minimal reliance on external knowledge. The integration of automated image generation with a training-free cropping method is novel, enabling scalable evaluation and enhancement of model perception. Unlike prior datasets limited to simple OCR or scientific reasoning, LogicOCR covers diverse real-world scenarios and reasoning types, pushing the boundary of multimodal understanding. The use of attention-guided, automatic region cropping as a perception enhancement is a key innovation, offering a practical, training-free way to improve model performance.
Limitations
- Models exhibit poor robustness under extreme rotations, occlusions, or complex backgrounds, limiting real-world applicability. The cropping method, while effective, may misidentify key regions in highly cluttered scenes, affecting accuracy.
- The evaluation focuses on specific models and scenarios, which may not generalize across all practical applications. Computational costs for image generation, verification, and evaluation remain high, hindering large-scale deployment.
- The current approach does not fully address the integration of reasoning paths, and models still struggle with multi-hop, multi-step inference in dense text scenarios. Future work should explore more adaptive, context-aware localization and reasoning strategies.
Future Work
Future directions include integrating multi-task learning to jointly improve OCR robustness and reasoning, expanding the benchmark to cover more real-world domains like medical and financial documents, and developing more efficient, adaptive localization algorithms. Additionally, combining this approach with pretraining on larger, domain-specific datasets could further enhance model capabilities. Exploring multimodal reasoning with less reliance on synthetic data and more on real-world annotated datasets will be crucial for practical deployment. The ultimate goal is to create models that can seamlessly read, understand, and reason over complex visual-text data in diverse environments.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器和工人。每个工人都在做不同的任务，比如组装零件、检查产品或整理材料。有时候，你需要找到某个特别的零件，或者确认某个流程是否正确。传统的方法可能需要你逐一检查每个部分，但如果你有一台聪明的机器人，它可以自动帮你找到关键的零件，甚至理解整个流程。这个机器人就像论文中的大模型，它通过学习大量图片和文字，变得越来越聪明。研究人员用自动生成的图片和问题，教它如何在复杂场景中找到答案，就像工厂用机器人帮忙检查生产线一样。这样，机器人就能更快、更准确地理解图片中的内容，帮我们处理各种复杂任务，比如自动识别文件、理解图表内容等。
ELI14 Explained like you're 14
想象你有一个超级聪明的朋友，他可以用眼睛看懂一堆图片，然后用脑袋告诉你里面写了什么或者发生了什么。有时候图片里的文字很小或者被遮挡，他就会迷糊。这篇论文就像在教这个朋友怎么变得更聪明。科学家们用电脑自动生成很多带文字的图片，让它们看起来像真实场景，然后让电脑学习怎么找到图片中的重要部分。接着，他们还教电脑用一种特别的方法，把图片中最重要的区域裁剪出来，放大一点，让电脑更清楚地看懂。这样，电脑就能更好地理解复杂的图片，回答各种问题。就像你用放大镜看细节一样，这个方法让电脑变得更聪明、更厉害！
Abstract
Recent advances in Large Multimodal Models (LMMs) have revolutionized their reasoning and Optical Character Recognition (OCR) capabilities. However, their complex logical reasoning performance on text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 2780 questions with two subsets, i.e., LogicOCR-Gen with 1100 multi-choice questions on generated images, and LogicOCR-Real with 1680 meticulously designed free-form questions on real-world images. For constructing LogicOCR-Gen, we first curate a text corpus from the Chinese National Civil Servant Examination, and customize an automatic pipeline to steer GPT-Image-1 to generate images with varied layouts and fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified. We evaluate a range of representative LMMs under Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. Moreover, we propose TextCue, a training-free method that enhances LMMs' perception of image regions containing important text cues for solving questions. We leverage LMMs' attention maps and an off-the-shelf text segmentation specialist to determine the region, which is then cropped and enlarged to augment the original image. Experiments show its effectiveness, e.g., a 1.8% accuracy gain over LLaVA-OV-1.5-8B under the CoT setting. Our benchmark is available at https://github.com/MiliLab/LogicOCR.