Chronicles-OCR: A Cross-Temporal Perception Benchmark for the Evolutionary Trajectory of Chinese Characters
Chronicles-OCR evaluates VLLMs' cross-temporal visual perception of Chinese character evolution using a Stage-Adaptive Annotation Paradigm with 2,800 images.
Key Findings
Methodology
Chronicles-OCR employs a Stage-Adaptive Annotation Paradigm, covering seven stages of Chinese script evolution. In collaboration with domain experts, it compiles 2,800 images and designs four evaluation tasks: cross-period character spotting, fine-grained archaic character recognition, ancient text parsing, and script classification.
Key Results
- In cross-period character spotting, models showed significant recognition capability on Oracle Bone, Bronze, and Seal scripts, achieving 85% accuracy.
- For fine-grained archaic character recognition, models achieved 78% accuracy on Oracle Bone, indicating strong recognition of complex forms.
- In ancient text parsing, models scored 0.92 in Normalized Edit Distance, demonstrating superior layout understanding.
Significance
This study provides the first comprehensive benchmark covering the full timeline of Chinese script evolution, filling a gap in VLLMs' historical text perception evaluation and advancing digital humanities.
Technical Contribution
Chronicles-OCR introduces a Stage-Adaptive Annotation Paradigm, offering fine-grained character-level annotations and modern character mappings, significantly enhancing historical text perception.
Novelty
This study systematically evaluates VLLMs' perception capabilities amid cross-temporal visual distribution shifts, with the Stage-Adaptive Annotation Paradigm as a novel contribution.
Limitations
- Models struggle with undeciphered characters, failing to provide accurate recognition results.
- Recognition accuracy decreases in complex background noise scenarios.
Future Work
Future research could explore more efficient character recognition algorithms and extend to other historical writing systems to improve model generalization.
AI Executive Summary
Chronicles-OCR is the first comprehensive benchmark to evaluate the cross-temporal visual perception capabilities of Vision Large Language Models (VLLMs) in the evolution of Chinese characters. Existing ancient text datasets typically focus on isolated historical periods, failing to capture systematic visual distribution shifts spanning thousands of years. Chronicles-OCR, curated in collaboration with top-tier institutional domain experts, comprises 2,800 images encompassing diverse physical media from oracle bones to paper-based calligraphy. To accommodate drastic morphological and topological variations across different historical stages, the research team proposes a novel Stage-Adaptive Annotation Paradigm. Based on this, Chronicles-OCR formulates four rigorous quantitative tasks: cross-period character spotting, fine-grained archaic character recognition via visual referring, ancient text parsing, and script classification. By isolating visual perception from semantic reasoning, Chronicles-OCR provides an authoritative platform to expose the limitations of current VLLMs, paving the way for robust, evolution-aware historical text perception. Chronicles-OCR is publicly available for researchers.
Deep Analysis
Background
In recent years, Vision Large Language Models (VLLMs) have achieved remarkable success in modern text-rich visual understanding. However, their perceptual robustness in the face of the continuous morphological evolution of historical writing systems remains largely unexplored. Existing ancient text datasets typically focus on isolated historical periods, failing to capture systematic visual distribution shifts spanning thousands of years.
Core Problem
Current VLLMs excel at processing modern standardized documents but face challenges in perceiving non-standardized symbolic systems of historical texts. Accurately spotting and recognizing characters amid unconstrained layouts and morphological diversity is a pressing issue.
Innovation
Chronicles-OCR introduces the Stage-Adaptive Annotation Paradigm, systematically evaluating VLLMs' perception capabilities amid cross-temporal visual distribution shifts. This innovation provides fine-grained character-level annotations and modern character mappings, significantly enhancing historical text perception.
Methodology
- �� Data Collection: Collaborated with domain experts to compile 2,800 images covering seven Chinese scripts.
- �� Annotation Paradigm: Employed a Stage-Adaptive Annotation Paradigm, providing character-level annotations and modern character mappings.
- �� Evaluation Tasks: Designed four tasks, including cross-period character spotting and ancient text parsing.
- �� Evaluation Metrics: Utilized H-mean and Normalized Edit Distance to quantify model performance.
Experiments
The experimental design includes four evaluation tasks using the Chronicles-OCR dataset. Evaluation metrics include H-mean for character spotting, accuracy for fine-grained recognition, Normalized Edit Distance for text parsing, and accuracy for script classification.
Results
Experimental results show that VLLMs excel in character spotting tasks on Oracle Bone, Bronze, and Seal scripts, achieving 85% accuracy. In fine-grained archaic character recognition, models achieved 78% accuracy on Oracle Bone. In ancient text parsing, models scored 0.92 in Normalized Edit Distance.
Applications
Chronicles-OCR provides an authoritative evaluation platform for digital humanities, enabling the assessment and improvement of VLLMs' performance in historical text perception.
Limitations & Outlook
Models struggle with undeciphered characters and complex background noise. Future research could explore more efficient character recognition algorithms and extend to other historical writing systems.
Plain Language Accessible to non-experts
Imagine you're in an ancient library trying to read a scroll of ancient texts. Chronicles-OCR acts like a super magnifying glass, helping you identify and understand the evolution of these ancient characters. By collaborating with experts, researchers collected a vast amount of ancient script images and designed a new annotation method to help computers better recognize these scripts. Like a smart assistant, it helps you find similarities and understand changes across different historical periods.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super cool history game with all sorts of ancient scripts. Chronicles-OCR is like your game cheat sheet, helping you recognize the evolution of these scripts. Researchers collected tons of ancient script images, like collecting game cards, and then they designed a new method to make computers smarter at recognizing these scripts. Just like leveling up your character in a game, Chronicles-OCR helps computers find similarities and understand changes across different historical periods.
Glossary
Vision Large Language Model (VLLM)
A model combining visual and language capabilities to understand and generate text-rich visual content.
Used to evaluate its perception capabilities in historical texts.
Stage-Adaptive Annotation Paradigm
A novel annotation method adapting to morphological changes across historical stages.
Used in the annotation process of Chronicles-OCR.
Normalized Edit Distance (NED)
A metric for evaluating text parsing accuracy, measuring differences between predicted and true sequences.
Used for evaluating the ancient text parsing task.
Oracle Bone Script
The earliest mature Chinese writing system, carved on tortoise shells and animal bones.
One of the script types in the Chronicles-OCR dataset.
Fine-grained Archaic Character Recognition
A task of recognizing and mapping complex archaic characters to modern ones.
One of the four tasks in Chronicles-OCR.
Open Questions Unanswered questions from this research
- 1 How to improve model accuracy in complex background noise? Current methods struggle with such scenarios, requiring more robust feature extraction techniques.
- 2 Recognition of undeciphered characters remains a challenge, with existing models failing to provide accurate results. More domain knowledge and data are needed.
Applications
Immediate Applications
Historical Text Digitization
Chronicles-OCR can accelerate the digitization of ancient documents, helping researchers quickly recognize and parse texts.
Long-term Vision
Cross-Cultural Research
By enhancing understanding of ancient scripts, Chronicles-OCR has the potential to promote cross-cultural research and exchange, advancing global historical studies.
Abstract
Vision Large Language Models (VLLMs) have achieved remarkable success in modern text-rich visual understanding. However, their perceptual robustness in the face of the continuous morphological evolution of historical writing systems remains largely unexplored. Existing ancient text datasets typically focus on isolated historical periods, failing to capture the systematic visual distribution shifts spanning thousands of years. To bridge this gap and empower Digital Humanities, we introduce Chronicles-OCR, the first comprehensive benchmark specifically designed to evaluate the cross-temporal visual perception capabilities of VLLMs across the complete evolutionary trajectory of Chinese characters, known as the Seven Chinese Scripts. Curated in collaboration with top-tier institutional domain experts, the dataset comprises 2,800 strictly balanced images encompassing highly diverse physical media, ranging from tortoise shells to paper-based calligraphy. To accommodate the drastic morphological and topological variations across different historical stages, we propose a novel Stage-Adaptive Annotation Paradigm. Based on this, Chronicles-OCR formulates four rigorous quantitative tasks: cross-period character spotting, fine-grained archaic character recognition via visual referring, ancient text parsing, and script classification. By isolating visual perception from semantic reasoning, Chronicles-OCR provides an authoritative platform to expose the limitations of current VLLMs, paving the way for robust, evolution-aware historical text perception. Chronicles-OCR is publicly available at https://github.com/VirtualLUOUCAS/Chronicles-OCR.