SEA-Embedding: Open and Reproducible Text Embeddings for Southeast Asia
SEA-Embedding employs contrastive learning and distribution matching, trained solely on public data, achieving SOTA in Southeast Asian multilingual embeddings.
Key Findings
Methodology
This paper introduces SEA-Embedding, combining Symmetric Contrastive Learning (SCL) and Similarity Distribution Matching (SDM) with publicly available datasets (245M text pairs, 14M instruction texts). The model initializes from various pre-trained encoders (e.g., E5-Large, ModernBERT), optimizing with temperature scaling and focal reweighting to enhance robustness. Large-scale memory queues are used for distribution alignment, fostering cross-lingual semantic consistency. The training pipeline ensures full reproducibility through open code, data, and environment setup.
Key Results
- On SEA-BED, SEA-Embedding scores an average of 0.800 across 10 Southeast Asian languages, surpassing models like multilingual-e5-large-instruct (0.789) and Qwen3-Embedding-8B (0.773). It significantly improves low-resource languages such as Lao and Khmer by over 20%, demonstrating robustness and regional adaptability.
- Ablation studies confirm data diversity (including non-SEA instruction data) and training objectives (notably LSDM) are critical. The model exhibits strong transferability across different pretraining initializations, with performance gains exceeding 0.3 points, validating the effectiveness of the proposed training recipe.
- Across multiple tasks and scenarios, SEA-Embedding maintains high consistency and robustness, illustrating the benefits of open, reproducible training strategies for regional NLP applications.
Significance
This work addresses the critical gap in open, reproducible, and region-specific multilingual embeddings for Southeast Asia. It advances the state-of-the-art by providing a transparent framework that enhances low-resource language performance, facilitating fairer AI development. The open-source approach fosters collaboration and accelerates progress in regional NLP, offering a practical solution to the resource scarcity challenge while setting a new standard for model transparency and robustness.
Technical Contribution
The core innovation lies in integrating symmetric contrastive learning with similarity distribution matching, leveraging large memory queues for global distribution alignment. The focus reweighting mechanism emphasizes hard examples, improving robustness. The framework's compatibility with various pre-trained encoders demonstrates its flexibility, providing a scalable, region-agnostic paradigm for multilingual embedding training. This approach advances theoretical understanding of cross-lingual semantic space construction and practical training stability.
Novelty
This is the first comprehensive, open, and reproducible framework tailored for Southeast Asian languages, combining dual-stage training objectives with publicly available data. The systematic analysis of data composition, training objectives, and model initialization offers new insights into robustness, filling a significant gap in regional NLP research. The integration of SDM with contrastive learning under an open setting is a novel contribution.
Limitations
- The model heavily relies on large-scale public datasets, which may not cover all domain-specific or dialectal variations, limiting performance in niche applications.
- Training costs are substantial, especially for larger models, restricting deployment in resource-constrained environments.
- The approach's effectiveness on dialectal, spoken, or code-switched language variants remains to be validated, necessitating further research.
Future Work
Future directions include incorporating multimodal data (e.g., speech, images) to enhance regional dialect understanding, integrating domain-specific datasets for specialized applications, and optimizing training efficiency for broader deployment. Further exploration of dialectal and colloquial language adaptation, as well as real-world downstream tasks, will be prioritized to expand the model's applicability.
AI Executive Summary
The linguistic diversity of Southeast Asia presents unique challenges for natural language processing, especially in developing robust, open, and reproducible multilingual embeddings. Existing models often rely on proprietary data, limiting transparency and regional applicability. To address this, SEA-Embedding introduces a novel training pipeline that combines Symmetric Contrastive Learning (SCL) and Similarity Distribution Matching (SDM), trained exclusively on publicly available datasets covering 10 languages and multiple tasks.
This approach leverages large-scale data, including 245 million general text pairs and 14 million instruction-based samples, to foster a semantic space that captures regional linguistic nuances. By employing temperature scaling and focal reweighting, the model emphasizes hard examples, enhancing robustness across languages and tasks. The use of large memory queues for distribution alignment ensures global semantic consistency, a key factor in cross-lingual transferability.
Experimental results demonstrate that SEA-Embedding achieves an average score of 0.800 on the SEA-BED benchmark, outperforming existing models such as multilingual-e5-large-instruct and Qwen3-Embedding-8B. Notably, the model shows significant improvements in low-resource languages like Lao and Khmer, with performance gains exceeding 20%. Ablation studies confirm the importance of data diversity, training objectives, and model initialization, providing a clear recipe for future work.
This research not only advances academic understanding of multilingual embedding training but also offers practical tools for industry applications, including multilingual search, content moderation, and cross-cultural communication. Its open-source nature ensures transparency, reproducibility, and community engagement, setting a new standard for regional NLP development. Future efforts will focus on multimodal integration, domain adaptation, and efficiency improvements to further expand the impact of this work.
Deep Dive
Abstract
Text embeddings are fundamental to many downstream applications, making robustness important for real-world NLP. However, most recent state-of-the-art embedding models are not reproducible because they rely on closed or undisclosed training data, and they remain insufficiently robust for Southeast Asian languages. We present SEA-Embedding, a fully open and reproducible text-embedding pipeline for Southeast Asian languages trained only on publicly available data, and use it to study three core factors of robust embedding design: data composition, training objective, and base encoder initialization. SEA-Embedding achieves state-of-the-art results on SEA-BED while enabling systematic and reproducible analysis of robust text embeddings for the region.