DeepSeek-OCR 2: Visual Causal Flow
DeepSeek-OCR 2 uses DeepEncoder V2 for visual causal flow, achieving a 3.73% performance boost on OmniDocBench v1.5.
Key Findings
Methodology
DeepSeek-OCR 2 employs DeepEncoder V2, combining visual tokens and causal flow queries. A custom attention mask enables dynamic reordering of visual tokens. Visual tokens use bidirectional attention, while causal flow queries use causal attention, with only causal flow query results being output.
Key Results
- DeepSeek-OCR 2 achieved 91.09% performance on OmniDocBench v1.5, a 3.73% improvement over DeepSeek-OCR.
- Reading order edit distance decreased from 0.085 to 0.057 with DeepSeek-OCR 2.
- In text recognition edit distance, DeepSeek-OCR 2 outperformed its predecessor in most document types.
Significance
This study introduces causal reasoning in vision-language models, simulating the logical sequence of human visual processing. It addresses the issue of fixed visual token order in traditional models, achieving significant progress in understanding complex document layouts and offering new insights for unified multimodal encoding.
Technical Contribution
DeepSeek-OCR 2's technical contribution lies in its innovative DeepEncoder V2 architecture, which enables dynamic reordering of visual tokens through causal flow queries, breaking the limitations of fixed order in traditional vision-language models. It also showcases the potential of language model architectures in visual encoding.
Novelty
DeepSeek-OCR 2 is the first to apply causal reasoning to dynamically reorder visual tokens, offering a processing method closer to human vision compared to traditional fixed-order methods.
Limitations
- Performance drops significantly when handling text-dense documents like newspapers, possibly due to the lower upper limit of visual tokens.
- Insufficient newspaper samples in training data affect model performance on this document type.
Future Work
Future work could increase the upper limit of visual tokens to improve recognition of text-dense documents and expand newspaper samples in training data. Exploring longer causal flow queries for multi-hop reordering is also a viable direction.
AI Executive Summary
DeepSeek-OCR 2 addresses the fixed visual token order issue in traditional vision-language models by introducing DeepEncoder V2. This model simulates the causal reasoning ability of human vision, dynamically reordering visual tokens based on image semantics to enhance understanding of complex document layouts.
DeepEncoder V2 achieves dynamic reordering of visual tokens through a custom attention mask, where visual tokens use bidirectional attention and causal flow queries use causal attention. This design allows semantic ordering of visual tokens without changing their count, achieving a processing method closer to human vision.
Experimental results on OmniDocBench v1.5 show that DeepSeek-OCR 2 improves overall performance by 3.73% over its predecessor, with notable improvements in reading order and text recognition. While there is room for improvement in handling text-dense documents, the model offers new insights for unified multimodal encoding.
Deep Analysis
Background
In recent years, vision-language models (VLMs) have made significant advances in joint understanding of images and text. However, traditional models often use a fixed visual token order, which contradicts the flexible processing of human vision. DeepSeek-OCR 2 introduces causal reasoning to simulate human visual processing, providing a new solution for understanding complex document layouts.
Core Problem
Traditional vision-language models often use a fixed order of visual tokens when processing complex layouts, ignoring the importance of semantic relationships in images. How to achieve dynamic reordering of visual tokens to better simulate human visual processing is a pressing issue.
Innovation
DeepSeek-OCR 2 achieves dynamic reordering of visual tokens through DeepEncoder V2. • Uses causal flow queries for semantic ordering of visual tokens. • Custom attention mask enables bidirectional attention for visual tokens and causal attention for causal flow queries. • Only outputs causal flow query results, enhancing the model's semantic understanding.
Methodology
- �� Visual tokens are initially processed by a vision tokenizer. • DeepEncoder V2 achieves dynamic reordering of visual tokens through a custom attention mask. • Visual tokens use bidirectional attention, while causal flow queries use causal attention. • Only causal flow query results are output for use by the language model decoder.
Experiments
Experiments were conducted on OmniDocBench v1.5, covering 9 major document types. The upper limit of visual tokens used was 1120, with evaluation metrics including overall performance, text recognition edit distance, and reading order edit distance. Comparisons with DeepSeek-OCR validated the effectiveness of the new model.
Results
DeepSeek-OCR 2 achieved 91.09% performance on OmniDocBench v1.5, a 3.73% improvement over DeepSeek-OCR. Reading order edit distance decreased from 0.085 to 0.057, showing significant improvement.
Applications
DeepSeek-OCR 2 can be used for OCR tasks in complex documents, especially in scenarios requiring high-precision text recognition and reading order understanding. Its dynamic reordering capability makes it valuable for generating training data for multimodal models.
Limitations & Outlook
While DeepSeek-OCR 2 performs well on most document types, its performance drops significantly when handling text-dense documents like newspapers. This may be due to the lower upper limit of visual tokens and insufficient newspaper samples in training data. Future improvements could include increasing the upper limit of visual tokens and expanding training data.
Plain Language Accessible to non-experts
Imagine you're organizing a complex photo album with various photos and captions. Traditional methods would have you flip through each page in order, but DeepSeek-OCR 2 acts like a smart assistant, rearranging them based on content and importance, making it easier to find what you're looking for. It's like having a friend who understands priorities, helping you quickly find the most relevant information.
ELI14 Explained like you're 14
Hey kiddo! Imagine you're playing a puzzle game, and the traditional way is to piece it together in order. But DeepSeek-OCR 2 is like a super-smart puzzle helper that rearranges the pieces based on patterns and colors, helping you finish the puzzle faster. Isn't that cool? That's how it works when handling complex documents!
Glossary
Visual Token
A small segment of an image representing a portion of its information.
In DeepSeek-OCR 2, visual tokens represent the basic units of an image.
Causal Flow Query
A mechanism for dynamically reordering visual tokens, mimicking human visual causal reasoning.
In DeepEncoder V2, causal flow queries are used to reorder visual tokens.
Custom Attention Mask
A mask used to control attention mechanisms, determining which tokens can attend to each other.
In DeepEncoder V2, custom attention masks enable bidirectional attention for visual tokens and causal attention for causal flow queries.
OmniDocBench v1.5
A benchmark dataset for evaluating document recognition performance, containing various document types.
DeepSeek-OCR 2 is evaluated on OmniDocBench v1.5.
Edit Distance
A metric for measuring the difference between two strings, indicating how many edits are needed to transform one string into another.
Used in document recognition to evaluate the accuracy of text recognition and reading order.
Open Questions Unanswered questions from this research
- 1 How to improve recognition performance for text-dense documents without increasing computational complexity?
- 2 How to extend the capabilities of causal flow queries to support more complex visual tasks?
Applications
Immediate Applications
Document OCR Service
DeepSeek-OCR 2 can be used in online OCR services to provide high-precision document recognition and reading order understanding.
Long-term Vision
Multimodal Model Training
By generating high-quality training data, it supports unified encoding and training of multimodal models.
Abstract
We present DeepSeek-OCR 2 to investigate the feasibility of a novel encoder-DeepEncoder V2-capable of dynamically reordering visual tokens upon image semantics. Conventional vision-language models (VLMs) invariably process visual tokens in a rigid raster-scan order (top-left to bottom-right) with fixed positional encoding when fed into LLMs. However, this contradicts human visual perception, which follows flexible yet semantically coherent scanning patterns driven by inherent logical structures. Particularly for images with complex layouts, human vision exhibits causally-informed sequential processing. Inspired by this cognitive mechanism, DeepEncoder V2 is designed to endow the encoder with causal reasoning capabilities, enabling it to intelligently reorder visual tokens prior to LLM-based content interpretation. This work explores a novel paradigm: whether 2D image understanding can be effectively achieved through two-cascaded 1D causal reasoning structures, thereby offering a new architectural approach with the potential to achieve genuine 2D reasoning. Codes and model weights are publicly accessible at http://github.com/deepseek-ai/DeepSeek-OCR-2.