Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
This study systematically compares BPE and Unigram-LM tokenizers on fixed 165-token chemical bases, revealing near-disjoint vocabularies and emphasizing tokenizer choice as a key model design decision.
Key Findings
Methodology
The paper conducts a controlled comparison of BPE and Unigram-LM on a fixed 165-character chemical glyph set across diverse, drug-like, and natural-product corpora, with vocabulary sizes of 256, 512, and 1024. By fixing the pre-tokenization boundary policy (cross-bracket or not) and corpus type, it isolates the influence of the subword algorithms. No downstream language models are trained; the evaluation focuses solely on the resulting tokenizers' vocabulary membership, granularity, and distribution. Metrics such as Jaccard overlap, token count increase, and frequency imbalance are used to quantify differences, ensuring the analysis reflects pure algorithmic effects.
Key Results
- Across all 22 matched conditions, the vocabularies built by BPE and Unigram-LM show minimal overlap (Jaccard index never exceeds 0.161, max 0.05 when weighted by frequency), indicating fundamentally different subword structures. Unigram-LM segments molecules into 29-41% more tokens, producing finer granularity. BPE's segmentation acts as a coarse approximation of Unigram-LM's, with 80-99% of molecules showing BPE's segmentation as a strict coarsening. These differences persist across corpus types, boundary policies, and vocabulary sizes, even at eight times larger vocabularies beyond the learnability threshold.
- The findings demonstrate that subword algorithm choice significantly shapes the tokenizer's structure, challenging the assumption of interchangeability inherited from natural language processing. The divergence remains stable and robust, emphasizing that tokenizer design is a critical modeling decision rather than a default setting.
Significance
This research highlights the structural differences in chemical SMILES tokenization driven by algorithm choice, questioning the universal application of natural language subword methods like BPE in chemistry. It underscores that tokenizer selection impacts model expressiveness and generalization, guiding future development of chemical language models. Recognizing the non-interchangeability of BPE and Unigram-LM in this domain informs better model architecture choices, ultimately improving applications such as drug discovery, reaction prediction, and molecular property modeling. The study advocates for tailored tokenization strategies aligned with chemical data characteristics, fostering more accurate and efficient AI-driven chemical research.
Technical Contribution
The paper pioneers a systematic, controlled comparison of BPE and Unigram-LM in a fixed chemical glyph base, quantifying their differences in vocabulary membership, granularity, and distribution. It introduces metrics like Jaccard overlap and frequency imbalance to characterize structural divergence, and demonstrates that these differences are stable across corpus types, boundary policies, and vocabulary scales. By confirming that subword algorithms are non-interchangeable design choices, it provides a foundational understanding that can inform future tokenizer development and chemical language modeling, emphasizing the importance of algorithm selection in model architecture.
Novelty
This is the first comprehensive, controlled study comparing BPE and Unigram-LM on a fixed, chemically grammatical glyph base across multiple corpora and vocabulary sizes. It reveals that the two algorithms produce fundamentally different subword vocabularies, a divergence that persists even at large vocabularies beyond the learnability threshold. The work challenges the default adoption of BPE inherited from NLP, positioning subword algorithm choice as a critical, non-trivial model design decision specific to chemical data, thus advancing the understanding of tokenizer impacts in cheminformatics.
Limitations
- The study does not evaluate downstream model performance, leaving open how vocabulary differences translate into predictive accuracy or generalization in practical tasks. Future work should incorporate model training and task-specific evaluation.
- Analysis is limited to fixed glyph bases and does not explore dynamic or adaptive subword learning, which could mitigate divergence or improve efficiency.
- Experiments focus solely on SMILES notation, excluding other chemical representations like SELFIES or InChI, which may exhibit different tokenization behaviors and should be examined in future research.
Future Work
Future directions include integrating these tokenizer insights into downstream models for property prediction, reaction modeling, and generative tasks, to assess the impact of subword choice on performance. Additionally, exploring adaptive or hybrid tokenization strategies tailored to chemical data, and extending analysis to alternative chemical representations, will deepen understanding and improve AI applications in chemistry.
AI Executive Summary
Chemical SMILES strings serve as a fundamental language for representing molecular structures, and the choice of tokenization method critically influences model performance. Traditionally, natural language processing techniques like Byte-Pair Encoding (BPE) have been adopted in cheminformatics, but their suitability remains unverified in the chemical domain. This study conducts a systematic, controlled comparison of BPE and Unigram-LM algorithms on a fixed 165-character glyph base, across diverse corpora including PubChem, ZINC-22, and COCONUT, with vocabulary sizes ranging from 256 to 1024. The results reveal that the two algorithms produce nearly disjoint subword vocabularies, with minimal overlap in membership and significant differences in segmentation granularity and token distribution. Unigram-LM tends to segment molecules into 29-41% more tokens, capturing finer structural details, whereas BPE's segmentation acts as a coarse approximation. These differences persist even at larger vocabularies, well beyond the threshold where embeddings are reliably learnable, indicating that the subword algorithm choice is a fundamental modeling decision rather than a default setting inherited from NLP. Importantly, the divergence in subword structures is stable across corpus types, boundary policies, and vocabulary scales, emphasizing that tokenizer design must be carefully considered in chemical language modeling. The findings challenge the assumption of interchangeability of BPE and Unigram-LM, advocating for tailored tokenizer strategies aligned with chemical data characteristics. This work provides a critical foundation for future research aiming to optimize chemical AI models, with implications for drug discovery, reaction prediction, and molecular property modeling. It underscores that tokenizer selection is not trivial but central to model architecture, and that understanding these structural differences can lead to more accurate and robust chemical AI systems. Future efforts should incorporate downstream task evaluations and explore adaptive tokenization approaches to further enhance chemical language understanding.
Deep Dive
Abstract
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.