LGAI-EMBEDDING-Preview Technical Report
Instruction-based text embedding framework using Mistral-7B, combining soft supervision and adaptive hard-negative mining for multi-task performance.
Key Findings
Methodology
The approach leverages a decoder-only large language model, Mistral-7B, guided by structured instructions and few-shot examples to generate context-aware embeddings across diverse tasks. It integrates in-context learning (ICL), where prompts encode task definitions, and employs soft supervision via relevance scores distilled from a dense retriever-reranker pipeline. These continuous relevance signals serve as fine-grained supervision, replacing traditional binary labels. Additionally, adaptive hard-negative mining filters out semantically ambiguous negatives based on similarity thresholds, enhancing training stability. The model uses a fusion of lexical, semantic, and reranker scores through Reciprocal Rank Fusion (RRF) to improve retrieval robustness. The training employs InfoNCE contrastive loss, with in-batch negatives and soft labels, without modifying model parameters, enabling zero-shot multi-task generalization.
Key Results
- On the MTEB-v2 benchmark with 41 tasks, the model achieves an average score of 74.12, with retrieval at 66.18, STS at 86.69, pair classification at 88.67, and summarization at 38.93, outperforming many larger or fully fine-tuned models.
- Compared to baselines, it maintains competitive performance without parameter updates, demonstrating effective in-context learning and soft supervision synergy.
- Adaptive margin-based hard-negative mining significantly improves training stability and retrieval accuracy, especially in semantically overlapping negatives.
Significance
This work advances the paradigm of leveraging large language models' inherent generalization abilities for multi-task text embedding without costly fine-tuning. It addresses the challenge of balancing flexibility, efficiency, and performance across diverse NLP tasks. By integrating structured prompts, soft relevance signals, and adaptive negative sampling, it provides a scalable solution suitable for real-world applications like search, recommendation, and semantic understanding, reducing deployment costs and enhancing adaptability.
Technical Contribution
The key innovation lies in combining in-context learning with soft supervision derived from dense retrieval pipelines, enabling zero-shot multi-task embedding. The use of adaptive margin-based negative filtering improves robustness against false negatives. The fusion of multiple retrieval scores via RRF enhances effectiveness. The approach demonstrates that large decoder-only models can serve as versatile, high-quality encoders without structural modifications, opening new avenues for scalable NLP systems.
Novelty
This is the first work to fully exploit large decoder-only LLMs' in-context capabilities for multi-task embedding without fine-tuning, integrating soft relevance supervision from dense retrievers, and employing adaptive negative filtering. It departs from traditional fine-tuning-based methods, emphasizing prompt engineering and soft supervision for scalable, high-performance embeddings.
Limitations
- The approach heavily depends on the quality of the dense retriever and reranker; if these components underperform, embedding quality declines.
- In scenarios with extreme semantic ambiguity or very few samples, soft supervision may lack sufficient granularity.
- Current method is primarily evaluated on English; multilingual extension remains to be explored.
Future Work
Future directions include expanding multi-language capabilities, integrating more sophisticated supervision signals, and optimizing retrieval pipelines for efficiency. Exploring continual learning and domain adaptation strategies will further enhance robustness and applicability in diverse real-world scenarios.
AI Executive Summary
The rapid development of large language models (LLMs) has revolutionized natural language processing, yet leveraging their full potential for multi-task text embedding remains challenging. Traditional methods rely heavily on fine-tuning, which is costly and inflexible. This paper introduces a novel instruction-based framework built upon the Mistral-7B model, designed to generate high-quality, context-aware embeddings across diverse NLP tasks without parameter updates.
The core innovation combines in-context learning (ICL), soft supervision from a dense retrieval pipeline, and adaptive hard-negative mining. Structured prompts and few-shot examples guide the model to understand task semantics implicitly. Soft labels, derived from relevance scores of a dense retriever and reranker, provide nuanced supervision signals that help the model distinguish subtle semantic differences. The adaptive negative mining filters out semantically ambiguous negatives based on similarity thresholds, improving training stability and robustness.
Experimental results on the newly introduced MTEB-v2 benchmark, covering 41 tasks including retrieval, classification, clustering, semantic similarity, and summarization, demonstrate the effectiveness of this approach. The model achieves an average score of 74.12, outperforming many larger or fully fine-tuned baselines. Notably, it ranks second in retrieval and STS, and first in summarization, showcasing its strong generalization and task adaptability.
This work highlights a scalable, efficient pathway for deploying multi-task NLP systems that leverage the inherent capabilities of large models. By avoiding costly fine-tuning, it opens new possibilities for real-world applications in search engines, recommendation systems, and semantic understanding. Future work will focus on multilingual extension, further refining supervision strategies, and optimizing retrieval components to broaden the impact of this promising approach.
Deep Dive
Abstract
This report presents a unified instruction-based framework for learning generalized text embeddings optimized for both information retrieval (IR) and non-IR tasks. Built upon a decoder-only large language model (Mistral-7B), our approach combines in-context learning, soft supervision, and adaptive hard-negative mining to generate context-aware embeddings without task-specific fine-tuning. Structured instructions and few-shot examples are used to guide the model across diverse tasks, enabling strong performance on classification, semantic similarity, clustering, and reranking benchmarks. To improve semantic discrimination, we employ a soft labeling framework where continuous relevance scores, distilled from a high-performance dense retriever and reranker, serve as fine-grained supervision signals. In addition, we introduce adaptive margin-based hard-negative mining, which filters out semantically ambiguous negatives based on their similarity to positive examples, thereby enhancing training stability and retrieval robustness. Our model is evaluated on the newly introduced MTEB (English, v2) benchmark, covering 41 tasks across seven categories. Results show that our method achieves strong generalization and ranks among the top-performing models by Borda score, outperforming several larger or fully fine-tuned baselines. These findings highlight the effectiveness of combining in-context prompting, soft supervision, and adaptive sampling for scalable, high-quality embedding generation.