Gecko: Versatile Text Embeddings Distilled from Large Language Models
Gecko employs a two-step distillation from LLMs, achieving 66.31 on MTEB with only 256 dimensions, outperforming larger models.
Key Findings
Methodology
This approach leverages large language models (LLMs) to generate diverse synthetic datasets. First, prompts create task-specific queries from web passages. Embedding these with a pre-trained model, the system retrieves candidate passages, which are then reranked by the LLM to select positive and hard negative samples. These high-quality pairs are combined with human-labeled data for multi-task contrastive training. The model, trained with a two-step distillation process, achieves state-of-the-art performance on the MTEB benchmark, with 256-dimensional embeddings surpassing larger models.
Key Results
- On MTEB, Gecko-256 outperforms all 768-dimensional models, achieving an average score of 66.31, demonstrating high efficiency and effectiveness. The 768-dimensional version reaches 66.31, rivaling models with 7x more parameters. Multilingual variants perform well across 18 languages, validating cross-lingual generalization. The synthetic data significantly boosts performance, especially in zero-shot settings, showing the power of LLM-guided data augmentation.
- In multilingual retrieval tasks, Gecko surpasses traditional methods like BM25 and mDPR, with notable improvements in nDCG@10 scores across diverse languages. Ablation studies reveal that combining synthetic and human data, along with reranking strategies, enhances robustness. The model balances retrieval, semantic similarity, and classification tasks, setting new benchmarks.
- The experimental results confirm that leveraging LLM-generated synthetic data and ranking improves low-dimensional models, reducing reliance on extensive labeled datasets while maintaining competitive performance across tasks.
Significance
This work addresses the bottleneck of data dependency in text embedding models by harnessing LLMs for synthetic data creation. The resulting compact models deliver high performance with fewer parameters, enabling scalable deployment in real-world applications such as search engines, question answering, and content recommendation. It bridges the gap between large, resource-intensive models and lightweight, versatile embeddings, fostering broader adoption in industry and research.
Technical Contribution
The paper introduces a novel two-step distillation framework that combines synthetic data generation, relevance ranking, and multi-task training. It innovates by integrating LLMs for both data synthesis and candidate reranking, enabling the training of small yet powerful models. The approach enhances robustness, cross-task generalization, and efficiency, providing a new paradigm for low-resource, high-performance text embeddings.
Novelty
This is the first comprehensive system to utilize LLMs for multi-task synthetic data generation, candidate reranking, and model training in a unified framework. Unlike prior work relying solely on manual annotations or limited synthetic data, this method leverages LLMs' knowledge and ranking capabilities to produce diverse, high-quality training pairs, significantly advancing the state-of-the-art in compact text embeddings.
Limitations
- The approach depends heavily on the quality of LLM outputs; biases or inaccuracies in LLM-generated data may affect model performance. Cross-lingual generalization beyond English remains to be validated, and the training process requires substantial computational resources due to multiple LLM calls.
- While effective in zero-shot settings, the model's performance in highly specialized or low-resource domains may decline without further adaptation. Scalability to extremely large datasets or real-time applications needs further optimization.
- Future work should focus on reducing LLM inference costs, enhancing multilingual capabilities, and addressing potential biases inherited from LLMs.
Future Work
Further research will explore multi-language synthetic data generation, adaptive training strategies, and more efficient LLM utilization. Developing methods for bias mitigation, expanding domain coverage, and deploying models in resource-constrained environments are also promising directions. Enhancing interpretability and robustness will be key to broader industrial adoption.
AI Executive Summary
In the rapidly evolving field of natural language processing, creating efficient yet powerful text embeddings remains a core challenge. Traditional models rely heavily on large annotated datasets, which are costly and limited in scope. Recent advances in large language models (LLMs) have opened new avenues for synthetic data generation, enabling the development of compact, multi-task capable embeddings.
This paper introduces Gecko, a novel text embedding framework that leverages LLMs to generate diverse synthetic datasets through a two-step process. First, prompts guide LLMs to produce task-specific queries from web passages, capturing a wide range of linguistic and semantic variations. These queries are embedded using a pre-trained model, and candidate passages are retrieved via similarity search. The LLM then reranks these candidates to identify positive and hard negative samples, ensuring high-quality training pairs.
Combining this synthetic data with human-labeled datasets, the authors train Gecko through multi-task contrastive learning. The key innovation lies in the integration of LLM-based relevance ranking and data synthesis, which significantly reduces the need for manual annotations. Experimental results on the MTEB benchmark demonstrate that a 256-dimensional Gecko model surpasses larger models with 768 dimensions, achieving an average score of 66.31. The model also exhibits strong multilingual performance, validating its generalization ability across languages.
This approach addresses critical issues in data dependency and model efficiency, paving the way for scalable deployment in real-world applications such as search engines, question answering, and content filtering. The authors highlight future directions including multilingual expansion, bias mitigation, and cost reduction, aiming to make high-performance text embeddings accessible and practical for industry use.
Deep Dive
Key Results
What do the experiments show?
Abstract
We present Gecko, a compact and versatile text embedding model. Gecko achieves strong retrieval performance by leveraging a key idea: distilling knowledge from large language models (LLMs) into a retriever. Our two-step distillation process begins with generating diverse, synthetic paired data using an LLM. Next, we further refine the data quality by retrieving a set of candidate passages for each query, and relabeling the positive and hard negative passages using the same LLM. The effectiveness of our approach is demonstrated by the compactness of the Gecko. On the Massive Text Embedding Benchmark (MTEB), Gecko with 256 embedding dimensions outperforms all existing entries with 768 embedding size. Gecko with 768 embedding dimensions achieves an average score of 66.31, competing with 7x larger models and 5x higher dimensional embeddings.