LangMAP: A Language-Adaptive Approach to Tokenization

The evolution of NLP has been driven by large-scale pretraining on massive corpora, leading to models like BERT, GPT, and T5 that rely heavily on effective tokenization. Traditional tokenizers such as BPE and UnigramLM excel in resource-rich languages but struggle with morphologically complex or low-resource languages, where over-segmentation and boundary misalignments are common. Multilingual models like mBERT and XLM-R attempt to unify multiple languages under a shared vocabulary, but often face trade-offs between universality and language-specific accuracy. Recent research explores vocabulary expansion, post-hoc adaptation, and character-level models, yet these solutions are computationally costly and often compromise model efficiency. The need for a scalable, language-adaptive tokenization method that maintains a shared vocabulary while providing language-specific segmentation remains unmet. This gap motivates the development of approaches like LangMAP, which leverages probabilistic modeling to reconcile these conflicting requirements.

The core problem addressed by this work is the suboptimal tokenization of diverse languages within a single shared vocabulary, leading to poor morphological boundary detection and downstream task performance. Existing models either require separate vocabularies for each language, incurring high storage and maintenance costs, or rely on post-hoc vocabulary modifications that are computationally expensive and inflexible. This results in a significant performance gap, especially for morphologically rich or low-resource languages, and hampers the scalability of multilingual NLP systems. Additionally, current methods lack an efficient mechanism to automatically identify the language of input text at inference, which is crucial for selecting the appropriate segmentation strategy. Overcoming these challenges requires a unified framework that can learn language-specific segmentation parameters without altering the shared vocabulary or retraining the entire model.

The primary innovation of this paper is the integration of language-specific unigram distributions within a shared vocabulary framework, enabling adaptive tokenization without vocabulary modification. This is achieved by:

• �� Learning separate unigram parameters ϕℓ for each language via EM, capturing language-specific morphological patterns;
• �� Employing a shared vocabulary to maintain model simplicity and facilitate transfer learning;
• �� Incorporating UniLID for automatic language detection during inference, allowing dynamic selection of the most probable segmentation model;
• �� Utilizing MAP estimation to efficiently compute the most likely segmentation under each language model, leveraging a Viterbi search over a fixed lattice.

This combination results in a scalable, flexible, and effective multilingual tokenization scheme that outperforms traditional methods in boundary alignment and downstream task accuracy.

• �� Training phase:
• �� Collect language-specific corpora, Cℓ, for each language ℓ.
• �� Initialize unigram distribution ϕℓ using empirical token frequencies.
• �� Run EM algorithm for 10 iterations:
• �� E-step: compute expected token counts over corpus Cℓ using current ϕℓ.
• �� M-step: normalize counts to update ϕℓ.
• �� Repeat until convergence, obtaining language-specific parameters.
• �� Inference phase:
• �� Use UniLID to identify input language.
• �� Construct a shared lattice of all possible tokenizations based on the fixed vocabulary.
• �� For each language ℓ, compute MAP segmentation likelihood using ϕℓ.
• �� Select the segmentation with the highest likelihood across all languages.
• �� Output the chosen segmentation as the final tokenization.
• �� Efficiency considerations:
• �� The lattice is built once, shared across languages.
• �� The per-language likelihood computations are linear in the number of languages, O(|Ω|).
• �� This design ensures scalable and fast inference suitable for large multilingual systems.

The experimental setup involved 14 open-source tokenizers, covering models from Gemma, LLama, Mistral, Qwen, and others, trained on datasets like FineWeb2 and The Stack. The evaluation spanned 9 natural languages, including German, Finnish, Turkish, Hindi, Bengali, Tamil, Arabic, and Thai, as well as 9 programming languages such as C, Java, and Python. For natural languages, corpora were drawn from Universal Dependencies treebanks, enabling morphological boundary evaluation via MorphScore. For code, parse trees generated by tree-sitter facilitated AST boundary analysis. Parameters for LangMAP were estimated with 10 EM iterations on 25 million tokens per language. The baseline was the original tokenizers, with comparisons based on intrinsic metrics (boundary alignment, entropy, compression) and extrinsic downstream tasks (grammar acceptability, reasoning, reading comprehension). Fine-tuning involved adapting pretrained models with LangMAP tokenization on 100 million tokens per language, assessing improvements in downstream performance.

LangMAP consistently improved morphological boundary alignment, with an average increase of +0.032 in MorphScore micro-F1 and +0.086 in recall across nine languages. In code, AST leaf boundary alignment improved by approximately 4.2%, indicating better structural tokenization. Fine-tuned models showed a 2-3.5% increase in grammatical acceptability scores on MultiBLiMP, especially for agglutinative languages like Turkish and Finnish. The improvements were less pronounced on reasoning and comprehension tasks, suggesting that morphological gains do not always translate directly to higher-level cognitive tasks. The results validate that language-specific parameter estimation enhances tokenization quality without retraining models, offering a practical solution for multilingual NLP pipelines.

This approach is immediately applicable to improving tokenization in multilingual language models, translation systems, and code understanding tools. It allows for rapid adaptation to new languages with minimal computational overhead, making it suitable for deployment in resource-constrained environments. Industry applications include multilingual chatbots, search engines, and automated content moderation, where accurate tokenization directly impacts performance. In the long term, LangMAP can facilitate the development of truly universal NLP systems capable of handling diverse linguistic and coding inputs seamlessly. Its scalability and efficiency also open avenues for real-time multilingual processing in voice assistants, real-time translation, and cross-modal AI systems, potentially transforming how machines understand and generate human language.

Despite its advantages, LangMAP faces limitations in languages with extremely limited corpora or poor script coverage, where the learned distributions may be unreliable. The automatic language identification step, while robust, can misclassify inputs with code-mixing or ambiguous scripts, leading to suboptimal segmentation. Computational costs, though linear, may become significant when scaling to hundreds of languages or dialects, especially in real-time applications. Additionally, the current framework assumes static corpora for parameter estimation, which may not adapt well to evolving language use or domain shifts. Future work should focus on unsupervised learning, dynamic adaptation, and integrating contextual cues to further enhance robustness and scalability.