BHARATI: Morphology-Aware Tokenizers for Classical Indian Languages with Subword Fertility Analysis
Proposes BHARATI, a morphology-aware tokenizer with subword fertility analysis, reducing sequence length by 90% on classical Indian language corpora.
Key Findings
Methodology
This work employs a SentencePiece-based BPE algorithm trained on a 781MB balanced multilingual corpus covering English, Hindi, Sanskrit, Tamil, Telugu, Kannada, and Malayalam. The development involved three versions: v1 (English, Sanskrit), v2 (adding Hindi, Tamil), and v3 (full seven-language native subword support). Key innovations include sandhi resolution for Sanskrit, suffix boundary marking for Tamil, and multi-language native subword entries. The evaluation used subword fertility metrics, measuring average tokens per domain-specific term, with v3 achieving 2.6 tokens per term, significantly lower than GPT-2's 5.25 and baseline SentencePiece's 3.75. Experiments on 490 IKS sentences demonstrated sequence length reductions of 90% relative to GPT-2 and byte-level encodings, and 25% compared to mBART-50, enhancing effective context length for downstream models.
Key Results
- v3 tokenizer reduces average tokens per IKS term to 2.6, compared to 5.25 with GPT-2, and 3.75 baseline, especially preserving single-token domain-specific terms.
- Sequence length on test data shrank by 90% relative to GPT-2, enabling models to process longer contexts efficiently.
- This efficiency gain directly improves training speed, model capacity, and downstream task performance in classical language NLP applications.
Significance
This research addresses the core challenge of effectively tokenizing morphologically complex and sandhi-rich classical Indian languages. By integrating morphological awareness into subword segmentation, it overcomes limitations of standard algorithms trained predominantly on modern languages. The resulting tokenizer enables more efficient language modeling, vital for educational tools, digital humanities, and cultural preservation. It paves the way for AI systems capable of understanding and generating classical texts with high fidelity, thus supporting the digital revival of India's linguistic heritage.
Technical Contribution
The paper introduces a multi-stage tokenizer development framework that incorporates sandhi resolution, suffix boundary marking, and multi-language native subword entries. The fertility analysis framework quantifies subword efficiency, guiding iterative improvements. The model employs a 32K vocabulary trained on a balanced corpus, ensuring high script and morphological coverage. This approach surpasses traditional frequency-based subword algorithms, offering a morphology-aware, domain-sensitive solution compatible with large-scale multilingual models.
Novelty
This is the first comprehensive integration of sandhi processing, suffix boundary marking, and multi-script native subword support within a single tokenizer tailored for classical Indian languages. Unlike prior work limited to modern scripts or monolingual settings, this approach explicitly models morphological phenomena unique to classical texts, achieving unprecedented subword efficiency and script coverage, thus bridging a crucial gap in multilingual NLP for morphologically rich languages.
Limitations
- Despite improvements, the tokenizer struggles with highly ambiguous sandhi junctions and nested compounds, leading to occasional segmentation errors.
- Limited classical corpus coverage for some rare or archaic forms affects the representation of certain domain-specific terms.
- The computational cost of training and inference remains high, necessitating further optimization for deployment in resource-constrained environments.
Future Work
Future directions include integrating deep morphological analyzers for more precise sandhi splitting, expanding corpus coverage with more classical texts, and exploring unsupervised learning techniques for further efficiency. Additionally, extending the framework to other morphologically complex languages and developing lightweight models for real-time applications are promising avenues.
AI Executive Summary
Classical Indian languages such as Sanskrit, Tamil, and Telugu possess intricate morphological structures, extensive sandhi phenomena, and multi-script representations, posing significant challenges for modern subword tokenization algorithms. Traditional methods like BPE and SentencePiece, trained predominantly on modern language corpora, often produce inefficient segmentations when applied to these classical languages, resulting in high subword fertility and elongated sequences that limit neural model performance.
To address these issues, the authors introduce BHARATI, a family of morphology-aware tokenizers built upon the SentencePiece BPE framework. The development process involved three iterative versions, progressively incorporating language-specific preprocessing steps such as sandhi resolution for Sanskrit, suffix boundary marking for Tamil, and native subword entries across all seven target languages. These innovations enable the tokenizer to produce more linguistically coherent subword units, significantly reducing the number of tokens needed to represent domain-specific terms.
Empirical evaluation on a carefully curated 781MB balanced corpus demonstrates that v3 achieves an average of 2.6 subwords per IKS term, a substantial improvement over baseline models. On a test set of 490 IKS sentences, the new tokenizer reduces sequence length by approximately 90% relative to GPT-2 and byte-level encodings, and by 25% compared to the mBART-50 baseline. These gains translate into increased effective context length, faster training, and improved downstream performance in language modeling tasks.
This work holds profound implications for computational linguistics and digital humanities, enabling more efficient and accurate processing of classical texts. It opens pathways for advanced AI applications in education, cultural preservation, and knowledge extraction from ancient manuscripts. Despite these advances, challenges remain in handling highly ambiguous morphological phenomena and optimizing computational efficiency. Future efforts will focus on integrating deep morphological analyzers, expanding corpus coverage, and refining models for deployment in resource-limited settings, further bridging the gap between classical linguistics and modern NLP technology.
Deep Dive
Abstract
Standard subword tokenization algorithms such as Byte-Pair Encoding (BPE) and SentencePiece are trained predominantly on modern language corpora and produce inefficient segmentations when applied to classical Indian languages. Sanskrit, Tamil, and other classical Indic languages exhibit agglutinative morphology, productive sandhi (phonological fusion at word boundaries), and domain-specific vocabularies absent from general-purpose training data. This paper presents BHARATI, a set of SentencePiece BPE tokenizers trained on a balanced 781 MB corpus spanning seven languages (English, Hindi, Sanskrit, Tamil, Telugu, Kannada, and Malayalam) with native script support for all languages. We describe three successive tokenizer versions: v1 (English and Sanskrit only, with broken byte-fallback for Tamil), v2 (four-language support with byte-level fallback for southern languages), and v3 (full seven-language native subword coverage). Subword fertility analysis demonstrates that v3 averages 2.6 tokens per Indian Knowledge System (IKS) technical term, compared to 5.25 tokens per term with GPT-2's tokenizer and 3.75 tokens with the multilingual SentencePiece baseline, with the largest gains on a set of reserved IKS terms that are represented as single tokens by construction. On a held-out test set of 490 IKS-domain sentences (70 per language across seven languages, released with the measurement script), v3 reduces sequence length by roughly 90% relative to GPT-2 and byte-level encoding (which lack native Indic subwords) and by approximately 25% relative to the mBART-50 multilingual baseline, averaged across the six Indic languages, directly translating to increased effective context length for downstream language models. The tokenizer models (32,000 vocabulary), training scripts, and evaluation benchmarks are released under open licenses.