LayoutLM: Pre-training of Text and Layout for Document Image Understanding
LayoutLM integrates text, layout, and visual features for document understanding, achieving 79.27% F1 on FUNSD.
Key Findings
Methodology
LayoutLM extends BERT by incorporating 2D position embeddings and image features, utilizing Faster R-CNN for visual extraction. The model employs a Masked Visual-Language Model (MVLM) and multi-label document classification during pretraining, enabling joint learning of textual, spatial, and visual information. This multi-task setup enhances understanding of complex document layouts, especially in scanned documents with diverse formats.
Key Results
- On the FUNSD form understanding dataset, LayoutLM achieved a 79.27 F1 score, surpassing BERT and RoBERTa by over 15%. In SROIE receipt extraction, accuracy increased from 94.02% to 95.24%. On RVL-CDIP document classification, accuracy rose from 93.07% to 94.42%. These improvements demonstrate the effectiveness of multimodal pretraining.
- Pretraining on over 6 million scanned documents, the model effectively captures spatial and visual cues, significantly boosting performance in downstream tasks. The integration of image features and layout information addresses limitations of text-only models, leading to more accurate information extraction.
- Experiments show that incorporating visual features benefits low-resource scenarios, with performance improving as data and training epochs increase. The model's robustness across tasks confirms its practical applicability in real-world document AI applications.
Significance
This work pioneers the joint pretraining of text, layout, and visual features in a single framework, addressing the limitations of prior models that focused solely on text. By capturing spatial relationships and visual cues, LayoutLM significantly advances document understanding, facilitating automated information extraction, form processing, and classification. Its ability to handle complex, diverse document formats makes it highly valuable for industry applications, reducing manual effort and increasing accuracy in enterprise workflows.
Technical Contribution
The paper introduces a novel multimodal architecture that fuses textual, spatial, and visual information within a unified pretraining framework. Key innovations include the 2D position embedding for spatial modeling and the integration of image features via Faster R-CNN. The multi-task training strategy, combining MVLM and multi-label classification, enhances the model's ability to generate rich, context-aware representations, setting a new standard for document AI models.
Novelty
This is the first work to jointly learn text, layout, and visual features in a single pretraining framework for document images. The combination of 2D spatial embeddings and visual features with a multi-task objective distinguishes it from prior single-modal or two-stage approaches, enabling comprehensive understanding of complex document layouts.
Limitations
- The model's performance heavily depends on OCR accuracy; errors in text recognition can impair downstream tasks, especially in low-quality scans.
- Pretraining large-scale models requires significant computational resources and time, limiting rapid deployment.
- Generalization to highly irregular or non-standard document formats remains challenging, necessitating further robustness improvements.
Future Work
Future directions include integrating end-to-end OCR and layout modeling, reducing reliance on external OCR quality. Efforts will focus on optimizing model efficiency, exploring multilingual and multi-format capabilities, and extending to handwritten and non-Latin scripts. Additionally, developing lightweight versions for real-time applications is a promising avenue.
AI Executive Summary
In the era of digital transformation, understanding complex documents automatically has become a critical challenge. Traditional NLP models excel at processing plain text but fall short when faced with scanned documents rich in layout and visual cues. Recognizing this gap, researchers led by Yiheng Xu introduced LayoutLM, a groundbreaking multimodal pretraining framework that jointly models text, spatial layout, and visual features.
LayoutLM builds upon the BERT architecture, enhancing it with 2D position embeddings that encode the relative spatial relationships among tokens, and image features extracted via Faster R-CNN. This integration allows the model to comprehend not only the textual content but also the layout and visual styles, such as fonts and colors, which are crucial for understanding complex documents like forms and receipts.
The training process employs a Masked Visual-Language Model (MVLM) objective, where tokens are randomly masked and predicted based on context, combined with a multi-label document classification task. This multi-task learning strategy enables the model to develop a holistic understanding of document structures. Pretrained on over 6 million scanned documents, LayoutLM demonstrated state-of-the-art results across multiple benchmarks, including 79.27% F1 on FUNSD, 95.24% accuracy on SROIE, and 94.42% on RVL-CDIP.
These results highlight the transformative potential of multimodal pretraining in automating document analysis tasks, reducing manual effort, and improving accuracy. The approach paves the way for more intelligent document AI systems capable of handling diverse and complex layouts, with broad applications in enterprise automation, legal processing, and information retrieval. Future work aims to optimize efficiency, extend multilingual support, and further integrate end-to-end OCR, promising a future where machines understand documents as intuitively as humans do.
Deep Dive
Abstract
Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation, while neglecting layout and style information that is vital for document image understanding. In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks, including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification (from 93.07 to 94.42). The code and pre-trained LayoutLM models are publicly available at \url{https://aka.ms/layoutlm}.