Goldfish: Monolingual Language Models for 350 Languages

The evolution of NLP has been marked by the development of large-scale multilingual models such as mBERT, XGLM, BLOOM, and MaLA-500, which aim to cover hundreds of languages simultaneously. These models, trained on massive datasets often exceeding terabytes, have achieved impressive results in high-resource languages like English and Chinese. However, their performance in low-resource languages—those with limited textual data—remains subpar, primarily due to data imbalance, script diversity, and model capacity constraints. Prior works have shown that monolingual models trained on dedicated datasets often outperform multilingual models in specific languages, especially when data is scarce. Despite this, the lack of publicly available monolingual models for many low-resource languages has hindered progress, creating a gap between research and practical application. Recent efforts to compile low-resource corpora, such as Glot500 and MADLAD, have begun addressing this gap, but systematic large-scale training and evaluation remain scarce. This background underscores the necessity for dedicated, scalable, and accessible monolingual models tailored for underrepresented languages, which can serve as foundational tools for linguistic preservation, cultural diversity, and equitable NLP development.

The core issue addressed in this research is the inadequacy of existing multilingual models in generating grammatically correct and contextually appropriate text in low-resource languages. These models, despite their size, often perform worse than simple statistical baselines like bigrams in perplexity metrics for many languages. The fundamental bottleneck lies in the limited amount of high-quality data, uneven data distribution, and the inability of large models to effectively learn language-specific nuances with scarce data. This results in poor linguistic representation, grammatical errors, and weak downstream task performance. Additionally, the absence of dedicated monolingual models for most low-resource languages hampers targeted research, model evaluation, and real-world applications. Addressing this problem requires developing models that can learn efficiently from minimal data, accurately capture language-specific features, and be openly accessible to the research community.

The key innovations of this work include: 1) a novel data scaling method using byte premium estimates to normalize dataset sizes across languages with diverse scripts and encoding efficiencies; 2) training of monolingual GPT-2 style models with custom Unigram tokenizers tailored to each language, ensuring efficient vocabulary usage; 3) systematic training of multiple model variants across different data scales (5MB, 10MB, 100MB, 1GB) to analyze the relationship between data volume and performance; 4) comprehensive evaluation against state-of-the-art multilingual models and simple baselines, demonstrating superior perplexity and grammaticality in low-resource settings. These innovations enable effective low-resource modeling by optimizing data utilization and model architecture for minimal data scenarios.

• �� Data collection: Merge corpora from Glot500, MADLAD, and other sources, deduplicate sequences, and exclude Bible-only datasets to ensure diversity.
• �� Byte premium estimation: Use Byte Premium tools to calculate language-specific byte overflow factors, normalizing dataset sizes across languages.
• �� Dataset sampling: Sample datasets to 5MB, 10MB, 100MB, and 1GB based on byte premium estimates, ensuring fair comparison.
• �� Tokenizer training: Train Unigram tokenizers with 50K vocabulary on each language’s dataset, preserving contiguity where possible.
• �� Model architecture: Use GPT-2 style autoregressive models with 125M parameters, trained on each dataset size, with hyperparameters optimized for low-resource conditions.
• �� Training process: Run 10 epochs per dataset, monitor perplexity and grammaticality metrics, and evaluate on FLORES-200 and linguistic benchmarks.
• �� Evaluation: Measure perplexity at sequence level, compare with multilingual models, and perform grammaticality tests using MultiBLiMP.
• �� Analysis: Conduct pairwise win rate comparisons, ablation studies on data size and model capacity, and error analysis to identify failure modes.

The experimental setup involves training models on datasets of varying sizes (5MB, 10MB, 100MB, 1GB) across 350 languages, with hyperparameters tuned for low-resource scenarios. Evaluation metrics include perplexity on FLORES-200, grammaticality accuracy on MultiBLiMP, and performance on multilingual reasoning benchmarks like Babel and XCOPA. Baselines include large multilingual models (XGLM, BLOOM, MaLA-500) and simple bigram models trained on the same data. The experiments aim to quantify the impact of data size and model capacity on linguistic performance, analyze language-specific results, and assess the models’ ability to generalize beyond training data. Cross-lingual transfer effects and robustness to data noise are also examined through ablation studies.

Results show that Goldfish models consistently outperform large multilingual models in perplexity for many low-resource languages, with an average perplexity reduction of 13% compared to XGLM 4.5B. In grammaticality tasks, Goldfish achieved an average accuracy of 78.8%, surpassing multilingual models and demonstrating strong linguistic understanding. Notably, in 43% of the languages tested, Goldfish models had lower perplexity than BLOOM 7.1B, despite being over ten times smaller. The models also showed robustness across different data scales, with diminishing returns observed beyond 1GB of data. However, performance on reasoning benchmarks remains weak, indicating the need for larger models or specialized training for complex tasks. The comparative analysis underscores the efficiency of dedicated monolingual models in low-resource contexts, emphasizing the importance of tailored data and architecture.

Immediate applications include low-resource language NLP tasks such as text generation, grammatical correction, and language preservation efforts. These models can be integrated into language-specific chatbots, translation tools, and educational resources, especially where data scarcity limits the use of large multilingual models. Long-term, the Goldfish framework can serve as a foundation for developing multilingual transfer learning techniques, enabling cross-lingual knowledge sharing while maintaining language-specific strengths. It also opens pathways for cultural preservation, digital inclusion, and the development of NLP tools for endangered languages, ultimately fostering linguistic diversity in AI applications.

While Goldfish models excel in basic text generation and grammaticality, their performance on complex reasoning and inference tasks remains limited, often near chance levels. The models are trained on a maximum of 1GB of data per language, which may not fully capture linguistic nuances or domain-specific knowledge. Data quality varies across sources, potentially introducing biases or noise. Additionally, the models are monolingual, which limits their ability to leverage cross-lingual transfer, and their small size constrains their capacity for sophisticated reasoning. Future work should focus on scaling models, enhancing data quality, and integrating multi-modal information to overcome these limitations.