MinGram: A Minimalist Unigram Tokenizer with High Compression and Competitive Morphological Alignment

Recent NLP advances rely heavily on subword tokenization techniques like BPE and Unigram. BPE's greedy merge approach is efficient but often produces fragmented or morphologically inconsistent tokens. Unigram, as a probabilistic model, better captures language structure but involves complex training steps, including suffix array mining, multiple EM iterations, and iterative pruning, which are computationally intensive. Recent research has sought to improve BPE's flexibility and Unigram's efficiency, but challenges remain in balancing compression, morphological alignment, and training complexity. The need for a simplified, scalable tokenizer that maintains linguistic quality is pressing, especially for multilingual and low-resource scenarios.

Traditional Unigram training relies on suffix array construction, multiple EM iterations, and iterative pruning, making the process slow and complex. BPE, while fast, often sacrifices morphological coherence and compression efficiency. The core challenge is to develop a tokenizer that combines the best of both worlds: high compression, morphological plausibility, and simple training. Achieving this requires rethinking the training pipeline to eliminate dependencies on suffix arrays and multi-round optimization, while ensuring the resulting tokens are linguistically meaningful and computationally efficient. This problem is critical for scaling large multilingual models and deploying NLP systems in resource-constrained environments.

The paper proposes three key innovations: 1) BPE-based seed vocabulary initialization, providing a compact starting point; 2) Hard EM optimization on the minimal token path, replacing traditional multi-path EM with a single, deterministic step; 3) a one-step flat score pruning, removing the need for multiple pruning cycles. These innovations collectively simplify the training pipeline, reduce computational overhead, and improve compression. The approach maintains Unigram's probabilistic scoring for tie-breaking, ensuring morphological relevance. This combination results in a highly efficient, scalable tokenizer that outperforms existing methods in both compression and morphological alignment.

• �� Initialize vocabulary with BPE trained on the corpus, using size \(\lceil f imes n ceil\).• Encode each sequence by selecting the minimal token path via a cost function balancing token count and Unigram score.
• �� Use Hard EM to update token probabilities based on the selected minimal path, avoiding soft EM's complexity.
• �� After EM iterations, perform a single flat score-based pruning to remove low-probability tokens, reaching the target vocabulary size.
• �� The training process is deterministic, with tie-breaking favoring longer leading tokens. The entire pipeline eschews suffix array mining, multiple EM cycles, and iterative pruning loops, focusing solely on inference-time efficiency.

Experiments involved six languages—English, German, Finnish, Russian, Arabic, Korean—using 5GB datasets for training and held-out corpora for evaluation. Baselines included BPE, standard Unigram, PathPiece, and ConvexTok. Metrics assessed were compression rate (tokens per character), morphological alignment (IBM1 score), and downstream bits-per-byte performance. Multiple hyperparameters were tested, especially the seed overshoot factor \(f\). Results consistently showed MinGram's superior compression and morphological quality, validated across multiple seeds and ablation studies. The evaluation also included downstream language modeling with 24-layer nanochat models, confirming practical benefits.

MinGram achieved 10-15% better compression than BPE and standard Unigram across languages. The MorphAlign score improved by over 20%, indicating better morphological coherence. In downstream tasks, MinGram outperformed BPE by approximately 8% in bits-per-byte, demonstrating its effectiveness in real-world applications. The variant MinGram-PP further enhanced compression while maintaining morphological alignment, showing the robustness of the pruning strategy. Ablation studies confirmed that the simplified training pipeline did not compromise performance, validating the design choices.

This approach is ideal for training multilingual large-scale models, especially where computational resources are limited. Its simplified pipeline reduces training time and hardware requirements, enabling faster deployment. The high compression and morphological fidelity benefit tasks like translation, speech recognition, and low-resource language modeling. Additionally, the method facilitates efficient model inference and storage, making it suitable for edge devices and real-time applications.

The current evaluation focuses on moderate-sized datasets and a limited set of languages, leaving questions about scalability to extremely low-resource or typologically diverse languages. Sensitivity to initial seed size requires careful tuning. The balance between compression and morphology still involves trade-offs, and further refinement of pruning strategies is needed to optimize both simultaneously. Future work should explore adaptive mechanisms and broader language coverage to address these limitations.