LangMAP: A Language-Adaptive Approach to Tokenization
LangMAP extends UnigramLM with language-specific MAP estimation, enabling multi-language tokenization from a shared vocabulary, improving morphological boundary alignment.
Clara Meister, Suchir Salhan, Andrzej Szablewski et al.