Enriching Word Vectors with Subword Information

The evolution of word embeddings has transitioned from count-based models to neural approaches like Mikolov’s word2vec, GloVe, and FastText. These models capture semantic relationships but often treat words as atomic units, neglecting internal morphology. For morphologically rich languages, this leads to data sparsity and poor generalization, especially for rare or unseen words. Prior solutions include morpheme-based models, factorization, and morphological segmentation, but they depend heavily on external analyzers or complex pipelines. Recent efforts like FastText incorporate subword information, yet often rely on fixed n-gram sets or external resources. The need for a simple, scalable, and language-agnostic method to incorporate subword structure remains pressing, motivating the development of character n-gram based models.

Traditional word embeddings struggle with morphological complexity, leading to poor representations of rare and unseen words. Existing methods often depend on external morphological analyzers, which are language-specific, or require complex segmentation, limiting scalability. This hampers NLP applications in low-resource languages and in tasks requiring robust out-of-vocabulary handling. The core challenge is to develop a model that captures internal word structure efficiently, generalizes well to unseen words, and remains computationally feasible across multiple languages and large datasets.

The paper's key innovations include: 1) Modeling words as bags of character n-grams, enabling parameter sharing and better morphological representation; 2) Extending skip-gram with negative sampling to optimize n-gram and word vectors simultaneously; 3) Using hashing to efficiently manage large n-gram vocabularies, reducing memory footprint; 4) Supporting out-of-vocabulary words by composing their vectors from known n-grams, improving robustness. These innovations collectively create a simple, effective, and scalable framework that captures subword information without external morphological tools, outperforming prior methods especially in morphologically complex languages.

• �� Extract character n-grams (n=3 to 6) from each word, adding boundary symbols
• �� Assign each n-gram a learnable vector, stored via hashing to manage size
• �� Represent each word as the sum of its n-gram vectors, including the full word
• �� Extend skip-gram with negative sampling: for each target word, predict context words and sample negatives
• �� Optimize the model using stochastic gradient descent, updating both word and n-gram vectors
• �� For unseen words, generate vectors by summing known n-gram vectors
• �� Conduct experiments on nine languages, tuning n-gram ranges and hyperparameters
• �� Evaluate on word similarity and analogy datasets, comparing with baselines
• �� Analyze the impact of n-gram length, training data size, and language morphology

The model was trained on Wikipedia dumps in nine languages, including morphologically rich ones like Russian and German. Evaluation involved word similarity tasks using datasets like WS353 and RW, and analogy tasks from Mikolov et al. The experiments varied n-gram lengths, training data sizes, and compared performance with traditional skip-gram and CBOW models. Additional tests assessed OOV handling, with ablation studies on n-gram range and hash size. Results confirmed the model's superior ability to represent rare and unseen words, especially in morphologically complex languages, with consistent improvements across tasks. Hyperparameters were tuned for optimal performance, and training efficiency was measured against baselines.

The character n-gram model achieved higher correlation scores on most word similarity datasets, with improvements of 5-10% in morphologically complex languages. In analogy tasks, syntactic accuracy improved by approximately 8%, demonstrating better grammatical relationship capture. The model maintained high performance for out-of-vocabulary words, with a notable ability to generate meaningful vectors from subword units even with limited training data. When trained on only 5% of the full corpus, it outperformed traditional models trained on full data, highlighting its robustness and efficiency. The impact was most pronounced in languages with rich morphology, validating the approach's effectiveness.

This method is ideal for multilingual NLP applications, including machine translation, information retrieval, and low-resource language processing. Its ability to generate embeddings for unseen words makes it valuable for real-time systems handling dynamic vocabularies. The approach can be integrated into pretraining pipelines, providing enriched representations that improve downstream task performance. It also supports rapid adaptation to domain-specific corpora, enabling domain-aware embeddings with minimal retraining. Future extensions could include contextual embeddings and hierarchical subword modeling, broadening its industrial relevance.

The choice of n-gram length (3-6) is heuristic and task-dependent, requiring tuning. Hash collisions may introduce noise, especially with smaller hash sizes, affecting embedding quality. In extremely low-resource settings, character n-grams alone may not fully capture semantic nuances, necessitating additional context modeling. The model's simplicity, while advantageous, may limit capturing complex morphological phenomena like reduplication or irregular forms. Future work should address these issues by adaptive n-gram selection and collision mitigation strategies.