Kleister: Key Information Extraction Datasets Involving Long Documents with Complex Layouts
Kleister datasets challenge existing KIE models; best model achieved 81.77% and 83.57% F1 on NDA and Charity datasets.
Key Findings
Methodology
The study uses a semi-supervised approach to collect datasets, combining textual and structural layout features for entity extraction. Baseline tests with models like Flair, BERT, RoBERTa, LayoutLM, and LAMBERT show the challenge posed by Kleister datasets.
Key Results
- On the Kleister NDA dataset, the best model LAMBERT achieved an F1 score of 81.77%, demonstrating its advantage in handling long documents and complex layouts.
- On the Kleister Charity dataset, the LAMBERT model achieved an F1 score of 83.57%, excelling in extracting income and spending entities from table structures.
- Experiments indicate that models using layout features have a significant advantage on complex layout documents.
Significance
This research fills a gap in long document and complex layout information extraction by providing new Kleister datasets, offering a new benchmark for academia and industry. It helps evaluate existing models' performance in real-world business scenarios, promoting progress in more complex information extraction tasks.
Technical Contribution
The study provides two new datasets, demonstrating the limitations of existing KIE models in handling long documents and complex layouts, and proposes models using layout features like LAMBERT to significantly improve performance.
Novelty
Introduces long document and complex layout KIE datasets for the first time, combining textual and structural layout features for entity extraction, offering higher challenge and practicality compared to existing datasets.
Limitations
- In some cases, models perform poorly on documents with low OCR quality, affecting entity extraction accuracy.
- Models have limited performance on entities requiring inference, such as monetary units.
Future Work
Future work can focus on improving OCR quality handling and developing more robust models for complex entity extraction tasks requiring inference.
AI Executive Summary
In the field of natural language processing, the task of Key Information Extraction (KIE) is becoming increasingly important, yet there is a lack of benchmark problems to evaluate solutions. To bridge this gap, researchers have introduced two new datasets: Kleister NDA and Kleister Charity. These datasets involve a mix of scanned and born-digital long English-language documents, requiring NLP systems to extract various entities using textual and structural layout features.
The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 pages and 21,612 entities to extract. The Kleister NDA dataset includes 540 Non-disclosure Agreements, with 3,229 pages and 2,160 entities to extract. Researchers provide several state-of-the-art baseline systems (e.g., Flair, BERT, RoBERTa, LayoutLM, LAMBERT) to demonstrate the strong challenge posed by these datasets to existing models.
In experiments, the best model achieved an F1 score of 81.77% and 83.57% on the Kleister NDA and Kleister Charity datasets, respectively. The researchers share these datasets to encourage progress on more in-depth and complex information extraction tasks.
Deep Analysis
Background
Key Information Extraction (KIE) is increasingly important in natural language processing, especially in the business market. Existing KIE systems perform well on publicly available datasets but still fall short in real-world applications. Recently, researchers have started filling this gap by creating new KIE datasets, such as scanned receipts and form understanding datasets.
Core Problem
Current KIE tasks lack benchmark datasets for long documents and complex layouts, limiting model evaluation and application in real-world business scenarios. Challenges include complex document layouts, specific business logic, and OCR quality, making information extraction tasks more difficult.
Innovation
The study introduces two new datasets, Kleister NDA and Kleister Charity, focusing on entity extraction from long documents and complex layouts. By combining textual and structural layout features, the study demonstrates existing models' performance on these datasets.
Methodology
- �� Use a semi-supervised approach to collect datasets, reducing manual annotation workload.
- �� Evaluate several state-of-the-art NER architectures (Flair, BERT, RoBERTa, LayoutLM, LAMBERT) using the Pipeline method.
- �� Conduct detailed analysis of data and baseline results.
Experiments
Experiments used the Kleister NDA and Kleister Charity datasets to evaluate models like Flair, BERT, RoBERTa, LayoutLM, and LAMBERT. The experimental design included dataset splitting, model training and evaluation, and OCR tool comparisons.
Results
Results show that models using layout features perform better on complex layout documents. The LAMBERT model achieved F1 scores of 81.77% and 83.57% on the Kleister NDA and Kleister Charity datasets, respectively.
Applications
The Kleister datasets provide a new benchmark for evaluating existing KIE models' performance on long documents and complex layouts. They can be used to develop more robust information extraction systems for business contracts, financial reports, and other scenarios.
Limitations & Outlook
The OCR tools used in the study have limited performance on low-quality scanned documents, affecting model accuracy. Some complex entity extraction tasks requiring inference remain challenging.
Plain Language Accessible to non-experts
Imagine you're in a library with many books, each with different layouts and content. Your task is to find specific information, like the author's name or publication date, from these books. The Kleister datasets are like these books, and our models are like a smart librarian who can quickly find the information you need, even if it's hidden in complex layouts.
ELI14 Explained like you're 14
Hey, imagine you're playing a game where you need to find hidden treasures on a bunch of complex maps. Each map has a different layout and clues. The Kleister datasets are like these maps, and our models are like a super detective who can help you quickly find the treasures! Isn't that cool?
Glossary
KIE (Key Information Extraction)
The process of extracting important information from documents, often involving entity recognition and relationship extraction.
In the paper, KIE is used to extract entities from long documents and complex layouts.
OCR (Optical Character Recognition)
A technology that converts text in scanned images into editable text.
Used to process scanned documents in the Kleister Charity dataset.
NER (Named Entity Recognition)
The process of identifying entities such as names of people, places, and organizations in text.
Various NER models were used in experiments for entity extraction.
LayoutLM
A model that combines text and layout information to process documents with complex layouts.
Used in experiments to evaluate its performance on the Kleister datasets.
LAMBERT
A model that combines text and layout features, specifically designed for processing documents with complex layouts.
The best-performing model in the experiments.
Open Questions Unanswered questions from this research
- 1 How to improve OCR tools' performance on low-quality scanned documents?
- 2 How to develop more robust models for complex entity extraction tasks requiring inference?
Applications
Immediate Applications
Business Contract Analysis
Models trained on the Kleister datasets can help businesses quickly analyze key information in contracts, improving efficiency.
Long-term Vision
Intelligent Document Management System
Develop a system that can automatically process and analyze documents with various complex layouts, enhancing enterprise information management capabilities.
Abstract
The relevance of the Key Information Extraction (KIE) task is increasingly important in natural language processing problems. But there are still only a few well-defined problems that serve as benchmarks for solutions in this area. To bridge this gap, we introduce two new datasets (Kleister NDA and Kleister Charity). They involve a mix of scanned and born-digital long formal English-language documents. In these datasets, an NLP system is expected to find or infer various types of entities by employing both textual and structural layout features. The Kleister Charity dataset consists of 2,788 annual financial reports of charity organizations, with 61,643 unique pages and 21,612 entities to extract. The Kleister NDA dataset has 540 Non-disclosure Agreements, with 3,229 unique pages and 2,160 entities to extract. We provide several state-of-the-art baseline systems from the KIE domain (Flair, BERT, RoBERTa, LayoutLM, LAMBERT), which show that our datasets pose a strong challenge to existing models. The best model achieved an 81.77% and an 83.57% F1-score on respectively the Kleister NDA and the Kleister Charity datasets. We share the datasets to encourage progress on more in-depth and complex information extraction tasks.