The Tatoeba Translation Challenge -- Realistic Data Sets for Low Resource and Multilingual MT

The evolution of machine translation has transitioned from statistical methods to neural models, notably Transformer architectures, which have achieved remarkable success in high-resource languages like English and French. However, low-resource languages lack sufficient data, limiting model performance and generalization. Prior datasets such as Europarl and WMT focus on well-resourced languages, neglecting the diversity and noise inherent in real-world data. OPUS provides a broad but uncurated collection, while Tatoeba offers crowd-sourced translations, capturing linguistic variability. Combining these sources, this work constructs a large, annotated, and realistic dataset, aiming to bridge the resource gap and promote inclusive multilingual translation.

The core challenge lies in developing effective translation models for languages with scarce data, often contaminated with noise, inconsistent scripts, and limited coverage. Existing benchmarks oversimplify low-resource scenarios, leading to overestimated transfer learning capabilities. Real-world conditions involve unbalanced, noisy, and diverse data, making it difficult for models to generalize. Zero-shot translation remains unreliable without sufficient cross-lingual transfer. Addressing these issues requires realistic datasets, robust models, and evaluation protocols that reflect actual language use, especially for endangered or underrepresented languages.

Key innovations include: 1) assembling a large-scale, authentic dataset with 555 languages and 2961 pairs, reflecting real-world data distribution; 2) automatic script detection and systematic cleaning to improve data quality; 3) designing multilingual Transformer models with language tags and shared vocabularies for effective transfer; 4) employing multi-reference and multi-task training to enhance robustness; 5) establishing standardized data splits and evaluation metrics for fair benchmarking. These innovations enable more realistic and scalable multilingual translation research, addressing the limitations of prior artificially balanced datasets.

• �� Data collection: Merge OPUS parallel corpora with Tatoeba crowd-sourced translations, ensuring broad language coverage.
• �� Data cleaning: Remove non-printable characters, normalize Unicode, restore special characters using Recode and Moses tools.
• �� Language detection: Apply CLD2 to identify language labels, filter out noise and non-target languages.
• �� Script annotation: Use Unicode properties and regular expressions to assign script codes, supporting multiple scripts per language.
• �� Data shuffling: Randomize data with Terashuf, maintaining source labels for traceability.
• �� Model training: Use Marian-NMT with Transformer architecture, 6 layers, 8 attention heads, SentencePiece tokenization, and language tags.
• �� Evaluation: Measure performance with chrF2 and BLEU across resource levels, compare single and multi-lingual models, perform ablation studies.

Experiments involve training models on datasets categorized by resource size—high (>10M pairs), medium (100K-1M), low (<100K), and zero-shot. Multiple language combinations are tested, including related and unrelated groups, to assess transfer effects. Hyperparameters like learning rate, batch size, and early stopping are tuned. The models are trained on V100 GPUs, with rigorous validation and testing. Ablation studies evaluate the impact of multi-reference data, language tags, and data filtering. Results are analyzed through chrF2 and BLEU scores, with detailed comparisons across resource levels and language groups, emphasizing the benefits of multilingual training.

Models trained on this dataset achieve chrF2 scores around 0.490 for high-resource pairs, with BLEU scores exceeding 50. Low-resource languages like Awadhi reach chrF2 0.285, demonstrating significant gains over prior approaches. Multilingual models outperform bilingual baselines in related language groups, e.g., Belorussian-English chrF2 0.610 and BLEU 42.7. Zero-shot translation for unseen languages like Awadhi yields chrF2 0.042, indicating room for improvement. The results confirm that large, diverse, and realistic datasets enable better transfer and generalization, especially in challenging low-resource scenarios.

The dataset and models support multilingual content translation, cross-lingual information retrieval, and digital preservation of endangered languages. Industry applications include deploying scalable translation services, enhancing global communication, and enabling multilingual chatbots. Educational tools can leverage these resources for language learning and cultural exchange. The models serve as a foundation for further research in unsupervised and semi-supervised learning, domain adaptation, and low-resource language technology, fostering inclusive digital ecosystems.

Despite extensive data, noise, imbalance, and script ambiguities persist, affecting low-resource model accuracy. Computational costs for training large multilingual models are high, limiting accessibility. Zero-shot performance remains weak, requiring more sophisticated transfer techniques. Data quality varies across sources, and some languages have minimal coverage, hindering scalability. Future work must focus on improving data curation, reducing training costs, and exploring unsupervised methods to better serve underrepresented languages.