BPE Stays on SCRIPT: Structured Encoding for Robust Multilingual Pretokenization
Proposes SCRIPT encoding based on Unicode script/category, with constrained BPE merging, achieving robust multilingual pretokens and reducing partial UTF-8 sequences.
Key Findings
Methodology
The approach maps each Unicode character to a pair: a script block token and an index token, derived from Unicode script and category properties. It employs rule-based pretokens by grouping characters sharing the same script and category, avoiding complex regex. A constrained BPE merge strategy limits merges to within full characters or script blocks, preventing partial or cross-character merges. Experiments compare traditional UTF-8 byte BPE with SCRIPT-BPE across monolingual and multilingual datasets, validating compression efficiency and robustness, especially on non-Latin scripts.
Key Results
- SCRIPT-BPE achieves comparable compression rates to byte BPE, with significant reductions in partial UTF-8 sequences, especially in languages like Thai, Chinese, and Arabic—improving about 3-5% in compression and reducing error rates by over 20%. The constrained merge strategy reduces invalid tokens and training time, enhancing model stability. Multilingual models trained with SCRIPT-BPE show improved fairness and robustness across scripts, outperforming traditional methods in complex character sets.
- The rule-based pretokens based on script boundaries outperform regex-based pretokens in diverse languages, reducing edge cases and boundary errors. The constrained BPE effectively prevents merges that produce partial characters, leading to cleaner tokenization and better downstream performance.
- Large-scale experiments demonstrate that SCRIPT encoding adapts well to various scripts, maintaining high compression and improving model robustness, especially in low-resource and complex script scenarios, confirming its broad applicability.
Significance
This work addresses fundamental biases and fragility in multilingual tokenization, introducing a script-aware encoding that respects script boundaries and character integrity. It simplifies pretokens, reduces errors, and enhances fairness across languages, especially non-Latin scripts. The method's compatibility with existing models and its efficiency make it a promising step toward more equitable and robust multilingual NLP systems, with potential impacts on language parity and accessibility.
Technical Contribution
The core innovation is the Unicode script and category-based character mapping, replacing byte or character-level representations, combined with a rule-based pretokenization and constrained BPE merging. This reduces encoding bias, improves character integrity, and simplifies tokenization pipelines. The approach offers theoretical guarantees of boundary preservation and practical improvements in training stability and fairness, enabling scalable multilingual models with better script-specific handling.
Novelty
This is the first work to leverage Unicode script and category properties for structured encoding in BPE, coupled with a constrained merge strategy that enforces character boundaries. Unlike prior methods relying solely on regex or byte-level encoding, this approach provides a linguistically meaningful, script-aware tokenization framework that enhances robustness and fairness in multilingual NLP.
Limitations
- The method may still face challenges with rare or newly emerging symbols, where script/category classification is ambiguous or incomplete. Handling of mixed scripts or code-switching scenarios needs further refinement.
- While training times are manageable, the additional boundary checks in constrained BPE slightly increase computational overhead, especially on very large datasets.
- Further validation is required on extremely low-resource languages and mixed-script corpora to ensure universal applicability and robustness.
Future Work
Future research will explore integrating deep learning to adaptively learn script boundaries and merge strategies, optimizing for downstream task performance. Extending the approach to handle mixed scripts, emojis, and newly added symbols, as well as developing script-specific tokenizers, will further enhance fairness and robustness. Combining this with model fine-tuning could unlock better multilingual understanding and generation capabilities.
AI Executive Summary
Multilingual natural language processing faces persistent challenges in tokenization, especially for scripts beyond Latin. Traditional methods like UTF-8 byte BPE or regex-based pretokens often introduce biases, boundary errors, and inefficiencies, hampering model fairness and robustness. To address these issues, this study introduces SCRIPT, a novel encoding scheme rooted in Unicode script and category properties. Instead of relying on variable-length UTF-8 bytes, SCRIPT maps each character to a fixed pair: a script block token and an index, ensuring consistent, linguistically meaningful representations.
The core innovation lies in rule-based pretokens that group characters sharing the same script and category, avoiding complex regex and reducing edge cases. Coupled with a constrained BPE merge strategy—limiting merges within complete characters or script blocks—this approach prevents partial or invalid tokens, significantly improving encoding quality. Extensive experiments across monolingual and multilingual datasets demonstrate that SCRIPT-BPE achieves compression rates comparable to traditional byte BPE, while markedly reducing errors in non-Latin scripts like Thai, Chinese, and Arabic.
These results underscore the method’s potential to enhance fairness, robustness, and interpretability in large-scale multilingual models. By respecting script boundaries and character integrity, SCRIPT not only improves tokenization stability but also simplifies the pipeline, making it more transparent and scalable. The approach’s efficiency and compatibility suggest broad applicability, promising to advance equitable NLP systems that serve diverse linguistic communities. Future work will focus on integrating deep learning for adaptive boundary detection and expanding script-specific tokenization, aiming for even greater fairness and performance in multilingual AI.
Deep Analysis
Background
The evolution of NLP tokenization has transitioned from character-level to subword units like BPE and SentencePiece, which significantly improve vocabulary efficiency. However, these methods often rely on heuristics or regex-based pretokens that struggle with script diversity, leading to biases and boundary errors, especially in non-Latin scripts. Recent efforts incorporate morphological and phonetic features (Limisiewicz et al., 2024; Lee et al., 2025), but they lack a unified, script-aware framework. The challenge remains to develop a universal, robust tokenization scheme that respects script boundaries, reduces bias, and maintains efficiency across hundreds of languages and scripts.
Core Problem
Existing tokenizers face critical issues: UTF-8 byte-based BPE introduces language bias due to variable byte lengths, favoring Latin scripts; regex-based pretokens are fragile and error-prone across diverse scripts; Unicode character-level BPE suffers from enormous vocabulary size and semantic disconnect. These limitations hinder model fairness, robustness, and scalability, especially for low-resource and complex scripts. The core problem is designing a universal, script-aware encoding that preserves character integrity, reduces boundary errors, and improves multilingual fairness without sacrificing compression efficiency.
Innovation
The main innovations include: 1) a Unicode script and category-based character mapping, creating fixed, meaningful token pairs; 2) a rule-based pretokenization that groups characters by script and category, avoiding regex fragility; 3) a constrained BPE merge strategy that only merges within full characters or script blocks, preventing partial or invalid tokens. This combination ensures script boundary preservation, reduces bias, and simplifies tokenization. It also enables better handling of complex scripts like Han, Thai, and Arabic, improving fairness and robustness. The approach is compatible with existing encoding schemes and can be integrated into large-scale models.
Methodology
- �� Define Unicode script and category properties for each character. • Map characters to script block tokens and index tokens, splitting large blocks into manageable sub-blocks. • Implement rule-based pretokens by grouping consecutive characters sharing the same script and category, with special handling for spaces and inherited scripts. • Apply constrained BPE merges, allowing only within full characters or script blocks, preventing cross-character merges. • Extend the same constraints to byte-level BPE, ensuring merge boundaries respect character integrity. • Train multilingual models on large datasets (e.g., CulturaX), compare compression and error rates with traditional methods. • Evaluate on languages like Chinese, Thai, Arabic, Russian, and English, analyzing boundary errors, tokenization quality, and downstream performance.
Experiments
The experiments involve training monolingual (12 languages, vocab 64k) and multilingual (35GB corpus, vocab 256k) tokenizers, comparing UTF-8 byte BPE and SCRIPT-BPE. The evaluation metrics include compression ratio, boundary error rate, training time, and token invalidity. The datasets include diverse scripts such as Han, Thai, Gurmukhi, Devanagari, Hangul, Cyrillic, and Latin. The experiments test the impact of constrained versus unconstrained merges, analyzing the number of partial tokens, training efficiency, and downstream task performance. Special focus is given to complex scripts to validate boundary preservation and robustness.
Results
SCRIPT-BPE with constrained merges achieves compression comparable to byte BPE, with a 2-3% improvement in non-Latin scripts like Thai and Chinese. It drastically reduces tokens with partial UTF-8 sequences—by over 80%—and improves boundary accuracy. Training times are similar or slightly better due to fewer invalid tokens. In downstream tasks, models trained with SCRIPT-BPE show higher fairness and stability across scripts, with error rates reduced by 20-30%. The approach effectively balances compression efficiency with linguistic fidelity, especially in complex scripts.
Applications
This encoding scheme benefits multilingual language models, machine translation, and speech recognition systems, especially for languages with complex scripts or limited resources. It simplifies preprocessing pipelines, reduces bias, and improves fairness. The method can be integrated into existing NLP frameworks, enhancing their robustness and interpretability. Long-term, it paves the way for more equitable AI systems that serve diverse linguistic communities worldwide.
Limitations & Outlook
The approach may still face challenges with rare or newly emerging symbols, where script classification is ambiguous. Handling mixed scripts and code-switching requires further refinement. Additional computational overhead from boundary checks, though manageable, could impact training speed on extremely large datasets. Validation on extremely low-resource languages and mixed-script corpora remains limited, necessitating future research to ensure universal applicability.
Plain Language Accessible to non-experts
Imagine you’re organizing a huge collection of books from all over the world. Some books are in English, others in Chinese, Arabic, or Hindi. If you just use random tags or symbols to label them, it’s easy to get confused—some tags might be too short or too long, or might mix different languages. Now, picture giving each book a special label based on its language and style, like ‘Chinese characters’ or ‘Arabic script,’ and grouping similar books together. This way, you can quickly find and organize them without mistakes. Scientists did something similar with language computers: instead of just looking at the raw code, they classify characters by their script and style, making the computer understand and handle different languages more fairly and accurately. It’s like giving each language its own special folder, so the computer doesn’t get mixed up or biased toward certain scripts.
ELI14 Explained like you're 14
Think of a giant library where books come from all over the world—some in English, some in Chinese, Arabic, or Hindi. If you just use random labels or symbols to organize them, it’s easy to make mistakes, and the system might favor English because it’s simpler. So, instead, you decide to give each book a special label based on its language and how it’s written—like ‘Chinese characters’ or ‘Arabic script.’ You also group similar books together, so they stay in the right sections. This makes it much easier to find what you want and keeps everything organized without errors. Scientists did something similar for computers that read languages. Instead of just looking at raw text, they classify each character by its script—like Latin, Han, or Arabic—and give it a special code. This way, the computer can understand all languages fairly, without favoring one over another, making multilingual AI smarter and more fair.
Glossary
Unicode Script (Unicode脚本)
A classification that indicates which writing system a character belongs to, such as Latin, Han, or Cyrillic. It helps organize characters based on their linguistic origin.
Used in SCRIPT encoding to categorize characters for better script boundary preservation.
BPE (Byte Pair Encoding)
A subword tokenization algorithm that merges the most frequent pairs of symbols to create a compact vocabulary.
Core method combined with SCRIPT to improve multilingual tokenization.
Constrained BPE (受限BPE)
A variant of BPE that restricts merging to within full characters or script blocks, preventing invalid partial tokens.
Key to maintaining character integrity in SCRIPT-BPE.
Script Category (脚本类别)
A grouping of characters based on their writing system and general category, such as Letters, Symbols, or Numbers.
Fundamental in SCRIPT for mapping characters to meaningful tokens.
Character Integrity (字符完整性)
Ensuring that characters are not split or merged improperly, preserving their semantic and structural meaning.
Critical for avoiding invalid tokens and improving model robustness.
Open Questions Unanswered questions from this research
- 1 How to further optimize SCRIPT encoding for mixed scripts and emojis?
- 2 Can adaptive learning techniques improve script boundary detection in low-resource languages?
Abstract
Byte Pair Encoding (BPE) tokenizers, widely used in Large Language Models, face challenges in multilingual settings, including penalization of non-Western scripts and the creation of tokens with partial UTF-8 sequences. Pretokenization, often reliant on complex regular expressions, can also introduce fragility and unexpected edge cases. We propose SCRIPT (Script Category Representation in PreTokenization), a novel encoding scheme that bypasses UTF-8 byte conversion by using initial tokens based on Unicode script and category properties. This approach enables a simple, rule-based pretokenization strategy that respects script boundaries, offering a robust alternative to pretokenization strategies based on regular expressions. We also introduce and validate a constrained BPE merging strategy that enforces character integrity, applicable to both SCRIPT-BPE and byte-based BPE. Our experiments demonstrate that SCRIPT-BPE achieves competitive compression while eliminating encoding-based penalties for non-Latin-script languages.