KinyaEmbed: Contrastive Sentence Embeddings for Kinyarwanda via Multi-Stage Curriculum Training
KinyaEmbed improves Kinyarwanda sentence embeddings via a four-stage curriculum, achieving Spearman ρ=0.7298.
Key Findings
Methodology
KinyaEmbed builds on KinyaBERT-large and employs a four-stage curriculum: Stage 1 uses 18,000 paraphrase pairs from the Official Gazette of Rwanda; Stage 2 fine-tunes on 715 MNLI-translated triplets; Stage 3 aligns embeddings with OPUS-100 English-Kinyarwanda pairs; Stage 4 refines using 2,936 high-quality pairs from KinyaCOMET.
Key Results
- Result 1: Achieved Spearman ρ=0.7298 on SemRel2024-rw, outperforming mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%.
- Result 2: On Wiki-RW-STS, Spearman ρ=0.6005, 8.6% higher than mE5-large-instruct.
- Result 3: Best document clustering Silhouette Score (0.2146) among all evaluated models.
Significance
KinyaEmbed is the first sentence embedding model tailored for Kinyarwanda, addressing the poor performance of multilingual models on this low-resource language. It enables semantic search, document clustering, and cross-lingual retrieval, crucial for Rwandan NLP.
Technical Contribution
Introduced a four-stage curriculum leveraging Kinyarwanda-specific pretraining and novel datasets like KinyaCOMET and Wiki-RW-STS. Demonstrated significant improvements over state-of-the-art multilingual models.
Novelty
This is the first model specifically designed for Kinyarwanda sentence embeddings, leveraging language-specific pretraining and curriculum training to outperform general-purpose multilingual models.
Limitations
- Limitation 1: Underperforms on asymmetric retrieval tasks compared to retrieval-optimized models.
- Limitation 2: FLORES-200 P@1 score is low (0.3587), far below translation-optimized models like LaBSE.
- Limitation 3: Evaluation datasets are primarily Wikipedia-based, lacking domain diversity.
Future Work
Future work could explore domain-specific datasets, improve cross-lingual retrieval, and incorporate instruction tuning for asymmetric tasks.
AI Executive Summary
Kinyarwanda, a low-resource Bantu language spoken by 12 million people, lacks robust NLP tools. Existing multilingual models like LaBSE and mE5 perform poorly due to limited representation in training data. KinyaEmbed addresses this gap by leveraging Kinyarwanda-specific pretraining and a four-stage curriculum training approach.
KinyaEmbed achieves state-of-the-art performance on Kinyarwanda sentence similarity tasks. It outperforms mE5-large by 20.9% on SemRel2024-rw and achieves a Spearman ρ of 0.6005 on the new Wiki-RW-STS benchmark. Additionally, it excels in document clustering, achieving the highest Silhouette Score (0.2146) among all evaluated models.
While KinyaEmbed underperforms on asymmetric retrieval tasks, it sets a new standard for Kinyarwanda NLP. Future work could focus on expanding domain-specific datasets and optimizing cross-lingual retrieval. By releasing the model and datasets, the authors provide a valuable resource for advancing low-resource language NLP research and applications in Rwanda.
Deep Analysis
Background
Kinyarwanda is a Bantu language spoken by 12 million people, primarily in Rwanda. Existing multilingual models like LaBSE and mE5 nominally support Kinyarwanda but perform poorly due to its underrepresentation in training corpora.
Core Problem
The core issue is the poor performance of multilingual models on Kinyarwanda sentence embeddings, which limits applications like semantic search and clustering. This stems from insufficient representation in pretraining data.
Innovation
KinyaEmbed introduces a four-stage curriculum: 1) monolingual paraphrase training; 2) MNLI-translated triplet fine-tuning; 3) cross-lingual alignment with OPUS-100; 4) refinement with high-quality KinyaCOMET pairs. This approach leverages Kinyarwanda-specific pretraining to address the language's unique challenges.
Methodology
- �� Stage 1: Fine-tune KinyaBERT-large on 18,000 paraphrase pairs from the Official Gazette of Rwanda.
- �� Stage 2: Use 715 MNLI-translated triplets for semantic reasoning fine-tuning.
- �� Stage 3: Align embeddings with OPUS-100 English-Kinyarwanda translation pairs.
- �� Stage 4: Refine embeddings using 2,936 high-quality pairs from KinyaCOMET.
Experiments
The model was evaluated on SemRel2024-rw and the newly constructed Wiki-RW-STS benchmark. Comparisons were made against LaBSE, mE5, and other multilingual models. Document clustering tasks further assessed embedding quality.
Results
KinyaEmbed achieved Spearman ρ=0.7298 on SemRel2024-rw, outperforming mE5-large by 20.9%. On Wiki-RW-STS, it achieved ρ=0.6005, 8.6% higher than mE5-large-instruct. It also achieved the best Silhouette Score (0.2146) in document clustering.
Applications
KinyaEmbed enables semantic search, document clustering, and cross-lingual retrieval for Kinyarwanda, making it suitable for resource-constrained environments like Rwandan government and community organizations.
Limitations & Outlook
The model underperforms on asymmetric retrieval tasks and has a low FLORES-200 P@1 score. Evaluation datasets are limited to Wikipedia, lacking domain diversity.
Plain Language Accessible to non-experts
Imagine organizing a multilingual library. KinyaEmbed acts as a specialized librarian for Kinyarwanda, expertly grouping related sentences and topics. Unlike general librarians (e.g., LaBSE), KinyaEmbed understands the unique nuances of Kinyarwanda, making it far more accurate.
ELI14 Explained like you're 14
Think of it like a language matching game! Other players use tools that don’t know Kinyarwanda well, so they mess up a lot. But KinyaEmbed is like a language expert—it quickly finds sentence matches and wins every round! Cool, right?
Glossary
KinyaBERT-large
A 12-layer Transformer model pretrained on Kinyarwanda text.
Used as the backbone for KinyaEmbed.
MultipleNegativesRankingLoss
A contrastive loss function for optimizing embedding similarity.
Applied in all training stages.
SemRel2024-rw
A Kinyarwanda semantic relatedness benchmark.
Used to evaluate STS performance.
KinyaCOMET
A high-quality Kinyarwanda-English sentence pair dataset.
Used in the final fine-tuning stage.
Wiki-RW-STS
A new STS benchmark derived from Kinyarwanda Wikipedia.
Used to test model generalization.
Open Questions Unanswered questions from this research
- 1 How can the model improve on asymmetric retrieval tasks?
- 2 Can larger cross-lingual datasets further enhance performance?
Applications
Immediate Applications
Semantic Search
Quickly find related content in Kinyarwanda Wikipedia.
Document Clustering
Organize government or educational materials by topic.
Long-term Vision
Cross-Lingual Bridge
Enable seamless retrieval between Kinyarwanda and English.
Abstract
We present KinyaEmbed, the first dedicated sentence embedding model for Kinyarwanda, a morphologically rich Bantu language spoken by over 12 million people in Rwanda. Existing multilingual embedding models such as LaBSE, mE5-large, and OpenAI text-embedding-3-large perform poorly on Kinyarwanda due to severe under-representation in their pre-training corpora. KinyaEmbed is built on KinyaBERT-large and trained via a four-stage curriculum using MultipleNegativesRankingLoss (MNRL): Stage 1 leverages ~18,000 paraphrase pairs from the Official Gazette of Rwanda with three temperature scales; Stage 2 fine-tunes on 715 NLLB-translated MNLI triplets for entailment structure; Stage 3 aligns representations using English-Kinyarwanda OPUS-100 translation pairs; Stage 4 refines with 2,936 high-quality pairs filtered from KinyaCOMET at quality threshold 0.8. We evaluate on SemRel2024-rw and introduce Wiki-RW-STS, a new contamination-free Kinyarwanda STS benchmark of 300 pairs derived from Kinyarwanda Wikipedia. A seven-checkpoint ensemble (all5+23A*2, with the final stage double-weighted) achieves Spearman \r{ho}=0.7298 on SemRel2024-rw, surpassing mE5-large by 20.9% and OpenAI text-embedding-3-large by 41.0%. KinyaEmbed also achieves the best document clustering silhouette score (0.2146) across all evaluated models. All checkpoints, the KinyaCOMET filtered pairs, and the Wiki-RW-STS benchmark are publicly available.