Conan-Embedding-v2: Training an LLM from Scratch for Text Embeddings
Conan-Embedding-v2 is a 1.4B parameter from-scratch trained model, using soft masks and dynamic hard negative mining to achieve SOTA text embeddings.
Key Findings
Methodology
Conan-Embedding-v2 employs from-scratch training of a 1.4B Transformer, integrating news and multilingual pair data for pretraining. Introduces a soft masking mechanism that smoothly transitions from causal to bidirectional masks, addressing training discrepancies. Utilizes a cross-lingual retrieval dataset covering 26 languages, enhancing multilingual fusion. Implements a dynamic hard negative mining strategy, continuously increasing negative sample difficulty to improve discriminative power. The training pipeline includes pretraining, supervised fine-tuning (SFT), weak supervision, and task-specific fine-tuning, combining InfoNCE and CoSENT losses for multi-task optimization.
Key Results
- Conan-Embedding-v2 achieves state-of-the-art results on MTEB and Chinese MTEB benchmarks, surpassing large models and specialized embedding models, with classification scores over 91 and average over 73. In zero-shot settings, it scores 73.52 on English MTEB, outperforming comparatives, demonstrating strong generalization.
- In cross-lingual retrieval, it attains top R@20 and nDCG@10 on MKQA, with improvements of +3.6% and +5.7%, indicating excellent multilingual alignment.
- Ablation studies confirm the effectiveness of soft masking and dynamic negative mining, with performance gains validating the design choices.
Significance
This work breaks through the limitations of fine-tuning-based embedding models by training from scratch, integrating innovative mechanisms for multilingual and multi-task learning. It significantly advances the state-of-the-art in multilingual semantic understanding, with broad implications for information retrieval, cross-lingual QA, and knowledge management, fostering more inclusive and efficient multilingual AI systems.
Technical Contribution
The paper introduces a from-scratch 1.4B Transformer, innovates with a soft mask scheduling mechanism for smooth transition between causal and bidirectional attention, and designs a cross-lingual dataset supporting 26 languages. The dynamic hard negative mining strategy maintains training challenge, boosting discriminative ability. The multi-stage training process effectively balances multi-task and multilingual requirements, outperforming existing fine-tuning approaches.
Novelty
This is the first work to incorporate a soft mask scheduling mechanism in large models to bridge causal and bidirectional training gaps from scratch. It also constructs a comprehensive cross-lingual retrieval dataset, enabling effective multilingual embedding fusion. These innovations collectively set a new paradigm for training efficient, high-performing multilingual text embeddings from scratch.
Limitations
- The model relies heavily on large-scale multilingual data, which may limit performance in low-resource languages. Data quality and coverage remain challenges.
- Soft mask scheduling involves hyperparameters requiring careful tuning, increasing training complexity.
- Performance in highly specialized or low-resource domains needs further improvement, and training costs remain high.
Future Work
Future directions include optimizing training efficiency, reducing data dependency, and extending multi-modal capabilities. Exploring adaptive mask scheduling, low-resource language support, and integrating visual or audio modalities could further enhance the model's versatility and real-world applicability.
AI Executive Summary
Conan-Embedding-v2 marks a significant advancement in multilingual text embedding. Unlike traditional approaches that fine-tune pretrained models, this work trains a 1.4B parameter Transformer from scratch, integrating diverse news and multilingual data to bridge data gaps. The core innovation is the soft masking mechanism, which allows the model to smoothly transition from causal to bidirectional attention, effectively addressing training discrepancies. This enables the model to learn comprehensive representations suitable for multiple tasks, including retrieval, classification, and semantic similarity, across 26 languages.
Simultaneously, the authors introduce a cross-lingual retrieval dataset that supports bidirectional search among 26 languages, significantly enhancing the model's multilingual alignment. To further improve discriminative ability, a dynamic hard negative mining strategy is employed, which continuously adapts the difficulty of negative samples during training. The entire process involves multiple stages—pretraining, supervised fine-tuning, weak supervision, and task-specific fine-tuning—optimized with InfoNCE and CoSENT losses.
Experimental results demonstrate that Conan-Embedding-v2 achieves state-of-the-art performance on both the MTEB and Chinese MTEB benchmarks, with classification scores exceeding 91 and average scores over 73. In zero-shot scenarios, it maintains strong generalization, outperforming larger models. The cross-lingual retrieval performance on MKQA also surpasses existing methods, confirming its multilingual capabilities.
This research offers a new paradigm for training efficient, high-quality multilingual text embeddings from scratch, with broad implications for information retrieval, cross-lingual understanding, and AI democratization. Future work will focus on reducing training costs, expanding low-resource language support, and integrating multimodal data, aiming to make multilingual AI more accessible and robust.
Deep Analysis
Background
随着大规模预训练模型的发展，文本嵌入技术已成为自然语言处理的核心。早期的Word2Vec、GloVe等方法实现了基本的词向量，后来BERT、RoBERTa等模型引入上下文信息，极大提升了表示能力。近年来，基于大模型的文本嵌入在信息检索、问答、语义匹配等任务中表现出色，但多依赖微调预训练模型，受限于数据差异和训练方式。微调方法如LoRA虽降低参数成本，但效果受限，特别在多语种场景中，模型难以融合不同语言的语义信息。本文试图从根本上解决这些问题，通过从零训练模型，结合丰富的多语种和新闻数据，推动多语种、多任务的统一表示。
Core Problem
现有文本嵌入模型多依赖微调预训练模型，面临数据鸿沟和训练差异的双重挑战。微调受限于预训练语料的局限性，难以实现跨任务和跨语言的高质量表示。多语种场景中，模型难以有效融合不同语言的语义信息，导致性能不平衡。此外，因果掩码和双向掩码的训练差异限制了模型在多任务中的表现，特别是在多语种、多任务环境下，模型的泛化能力不足。解决这些问题需要一种从根本上改变训练策略的方法，兼顾多语种、多任务、多场景的需求。
Innovation
本文的核心创新包括：1）从零训练1.4B参数Transformer模型，突破微调限制，提升模型自主学习能力；2）引入软掩码机制，实现因果掩码到双向掩码的平滑过渡，解决训练中的掩码类型差异；3）设计跨语言检索数据集，支持26语种的跨语义匹配，增强多语种融合能力；4）提出动态硬负样本挖掘策略，持续提升负样本难度，增强判别能力。这些创新结合，显著提升多语种文本嵌入的质量和泛化能力，推动模型在多任务、多场景中的应用。
Methodology
- �� 预训练阶段：采用新闻、问答、网页数据，训练8层、隐藏维度3584的Transformer模型，支持32,768 tokens输入。• 引入多语种配对数据，扩展训练语料，结合新闻和多语种数据弥合数据差距。• 软掩码机制：定义调度函数α(t)，逐步从0变到1，调节掩码类型，平滑过渡因果到双向掩码。• 跨语种检索数据集：采集26语种的检索对，翻译查询，支持跨语义匹配。• 动态硬负样本挖掘：在训练中实时检测负样本难度，替换不再困难的负样本，保持训练难度。• 多阶段训练：包括弱监督、微调、任务微调，结合InfoNCE和CoSENT损失优化多任务性能。
Experiments
使用MTEB和中文MTEB作为主要评估基准，比较多种模型性能。训练中调节参数，验证软掩码和负样本策略的效果。模型参数规模控制在1.4B，推理速度在合理范围内，确保实用性。通过消融实验验证各组件贡献，分析不同设置对性能的影响，确保模型在多任务、多语种环境中的优越表现。
Results
Conan-Embedding-v2在MTEB和中文MTEB上均达SOTA，分类任务得分超过91，重排序73，平均分超过73分。零样本环境中，英文表现73.52，优于对比模型。跨语言检索在MKQA上，R@20提升3.6%，nDCG@10提升5.7%。消融实验显示软掩码和硬负样本策略对性能提升显著，验证了设计的有效性。整体结果表明，从零训练结合创新机制能显著优于微调方案。
Applications
模型可广泛应用于多语种信息检索、跨语言问答、语义匹配等场景。支持多任务、多语种环境，适合构建多语言搜索引擎、多语种知识库等。未来结合多模态信息，有望实现更丰富的跨模态、多语种应用，推动国际化信息处理的发展。
Limitations & Outlook
模型依赖大量多语种数据，少数低资源语种表现仍有限。软掩码调度参数复杂，训练调优成本高。在极端低资源或专业领域，性能可能不足。未来需优化训练效率，扩展多模态能力，提升低资源语种表现。
Plain Language Accessible to non-experts
想象一个工厂里有很多不同的机器，每台机器都要学会识别各种不同的产品。以前，工厂会用专门的培训让每台机器只学会一部分产品，但这样效率很低。现在，这个新方法像是让所有机器从一开始就学会了所有产品的特征，而且还设计了一种特殊的调节机制，就像给机器逐步调整学习策略，让它们既能专注细节，又能看全局。这样，工厂的机器不但学得快，还能同时处理多种产品，更聪明、更灵活。这个方法还让不同国家的机器都能理解彼此的产品信息，就像让来自不同国家的工人都能用同一种标准识别商品一样。通过不断调整学习难度，工厂里的机器也变得越来越厉害，能应对各种复杂的任务。最终，这个系统可以用在搜索引擎、翻译、问答等很多地方，让信息变得更快、更准、更智能。
ELI14 Explained like you're 14
Imagine you're in school learning different languages and subjects. Before, teachers gave you a book and you memorized it step by step. But if you could understand many languages from the start and even help your friends translate, you'd be much cooler! That's what Conan-Embedding-v2 does: it learns to understand sentences in many languages from scratch, without relying on pre-made models. It uses a clever trick called 'soft masking'—think of it like gradually opening a window wider so you see more of the view. At first, it focuses on small parts, then slowly sees the whole picture, helping it learn better. It also challenges itself by picking harder examples during training, making it smarter over time. This way, it can do many tasks—finding information, answering questions, translating languages—all at once. It becomes a super helpful tool for anyone working with languages, making communication and information access much easier and faster. And as it keeps learning, it might even understand pictures or sounds someday, making it even more useful!
Glossary
Transformer
一种深度学习模型结构，擅长处理序列数据，广泛应用于自然语言处理。
模型架构基础，用于实现文本理解和生成。
软掩码机制
一种动态调节掩码类型的方法，使模型逐步从因果掩码过渡到双向掩码。
解决训练中掩码类型差异的问题。
跨语种检索数据集
包含多语种问答对，用于训练模型理解不同语言间的语义关系。
支持多语种表示融合。
InfoNCE损失
一种对比学习损失，用于最大化正样本间的相似度，最小化负样本间的相似度。
优化文本嵌入的相似性。
动态硬负样本挖掘
在训练过程中实时检测负样本难度，替换不再困难的负样本，保持训练挑战性。
提升模型判别能力。
Open Questions Unanswered questions from this research
- 1 如何进一步降低模型训练成本，提升效率，尤其是在低资源语种中实现高性能。
- 2 软掩码调度参数的最优设置及其对模型表现的影响机制尚不完全清楚。
- 3 多模态、多任务融合的潜力及其实现路径仍需深入探索。
Applications
Immediate Applications
多语种搜索引擎
利用模型实现跨语种信息检索，提高搜索的准确性和覆盖面，适合多语言国家和企业。
多语种问答系统
支持多语种用户提问，提供准确答案，提升国际化服务能力。
Long-term Vision
全球知识图谱
构建多语种、多模态的知识库，实现全球信息的无缝连接与智能管理。
Abstract
Large language models (LLMs) have recently demonstrated excellent performance in text embedding tasks. Previous work usually use LoRA to fine-tune existing LLMs, which are limited by the data and training gap between LLMs and embedding models. In this work, we introduce Conan-embedding-v2, a new 1.4B-parameter LLM trained from scratch and fine-tuned as a text embedder. First, we add news data and multilingual pairs for LLM pretraining to bridge the data gap. Based on this, we propose a cross-lingual retrieval dataset that enables the LLM to better integrate embeddings across different languages. Second, whereas LLMs use a causal mask with token-level loss, embedding models use a bidirectional mask with sentence-level loss. This training gap makes full fine-tuning less effective than LoRA. We introduce a soft-masking mechanism to gradually transition between these two types of masks, enabling the model to learn more comprehensive representations. Based on this, we propose a dynamic hard negative mining method that exposes the model to more difficult negative examples throughout the training process. Being intuitive and effective, with only approximately 1.4B parameters, Conan-embedding-v2 achieves SOTA performance on both the Massive Text Embedding Benchmark (MTEB) and Chinese MTEB (May 19, 2025).