Reasoning Quality Matters: Combating Reasoning Collapse in LLM-based Embedding Learning
CoFree framework addresses reasoning collapse in LLM embedding learning, achieving a 2.8 nDCG@10 improvement.
Key Findings
Methodology
The CoFree framework addresses reasoning collapse through a two-stage approach. The first stage uses reference-guided supervised fine-tuning to restore reasoning capabilities, while the second stage employs dual rewards in reinforcement learning to optimize embedding and reasoning quality. This ensures fine-grained relevance of reasoning and transforms embedding learning into a high-quality reasoning-guided search process.
Key Results
- CoFree-4B achieves a 2.8 nDCG@10 improvement over Qwen3-Embedding-4B across 22 datasets from MTEB and BRIGHT.
- Online experiments show consistent gains in a real-world retrieval system.
- Experiments validate CoFree's effectiveness in mitigating reasoning collapse across various frozen embedding backbones.
Significance
This research significantly enhances the reasoning quality in LLM embedding learning through the CoFree framework. It addresses the reasoning collapse issue, ensuring the relevance of generated reasoning and the discriminative power of embeddings, which is crucial for improving the accuracy and efficiency of information retrieval.
Technical Contribution
The CoFree framework balances reasoning generation and embedding discrimination. By employing reference-guided supervised fine-tuning and dual-reward reinforcement learning, it ensures the relevance of reasoning and the discriminative power of embeddings, surpassing existing SOTA methods.
Novelty
CoFree is the first to systematically address reasoning collapse in LLM embedding learning, introducing reference-guided supervised fine-tuning and dual-reward mechanisms to significantly enhance the relevance of reasoning generation.
Limitations
- Performance on code retrieval tasks is slightly inferior to some baseline models, possibly due to domain-specific reasoning needs.
- Requires substantial computational resources for training, limiting accessibility for smaller teams.
Future Work
Future work could explore applications on larger datasets and adaptability across different domains. Research could also focus on reducing computational costs to improve accessibility.
AI Executive Summary
Large Language Models (LLMs) have shown great potential in generating context-rich text embeddings, but specialization towards embedding objectives can lead to reasoning collapse. The CoFree framework addresses this issue through a two-stage approach. First, reference-guided supervised fine-tuning restores reasoning capabilities while retaining the representational strength of the foundational embedding model. Then, dual rewards in reinforcement learning optimize embedding and reasoning quality. Experiments demonstrate significant improvements in retrieval performance across multiple datasets, with consistent gains in online experiments. While there is room for improvement in specific tasks, CoFree provides a robust solution for LLM embedding learning.
Deep Analysis
Background
With the evolution of information retrieval technology, dense text embeddings have become a fundamental component of modern information retrieval. Traditional methods like TF-IDF and BM25, while efficient, fail to capture deeper semantics. The emergence of large language models (LLMs) offers new possibilities for embedding learning by enhancing semantic expression through reasoning text generation.
Core Problem
Existing methods may suppress useful reasoning generation or produce retrieval-irrelevant text when optimizing for embedding objectives, leading to reasoning collapse. This collapse affects the accuracy and efficiency of retrieval.
Innovation
The CoFree framework addresses reasoning collapse through a two-stage approach. The first stage uses reference-guided supervised fine-tuning to restore reasoning capabilities, while the second stage employs dual rewards in reinforcement learning to optimize embedding and reasoning quality.
Methodology
- �� Reference-guided supervised fine-tuning: Restores reasoning capabilities while retaining embedding representational strength.
- �� Dual reward mechanism: Optimizes embedding and reasoning quality in reinforcement learning, ensuring fine-grained relevance of reasoning.
Experiments
Experiments were conducted on 22 datasets from MTEB and BRIGHT, using nDCG@10 as the evaluation metric. Comparisons with multiple baseline models validated the effectiveness of CoFree.
Results
CoFree-4B achieves a 2.8 nDCG@10 improvement over Qwen3-Embedding-4B across multiple datasets. Online experiments show consistent gains in a real-world retrieval system.
Applications
CoFree can be directly applied to information retrieval systems to enhance retrieval accuracy and efficiency. It is suitable for scenarios requiring high-quality text embeddings, such as search engines and recommendation systems.
Limitations & Outlook
While CoFree performs excellently in multiple tasks, its performance on code retrieval tasks is slightly inferior to some baseline models. Additionally, the training process requires substantial computational resources, limiting accessibility for smaller teams.
Plain Language Accessible to non-experts
Imagine a library where CoFree acts like a smart librarian. Traditional methods simply arrange books alphabetically, while CoFree understands the theme and content of each book and recommends the most suitable ones based on the reader's needs. This way, CoFree ensures each recommendation is precise and relevant.
ELI14 Explained like you're 14
Imagine you're playing a massive multiplayer online game. There are many quests and clues, and traditional methods just look at the quest titles. CoFree, like a smart assistant, understands the background and details of each quest, helping you find the most relevant clues. This way, you can complete quests faster and earn more rewards!
Glossary
Reasoning Collapse
Refers to the degradation of reasoning generation capabilities when optimizing for embedding objectives, leading to irrelevant or inaccurate text generation.
In LLM embedding learning, reasoning collapse affects retrieval accuracy.
Reference-Guided Supervised Fine-Tuning
A fine-tuning method guided by a reference model to restore reasoning generation capabilities.
Used in the first stage of the CoFree framework to restore reasoning capabilities.
Dual Reward Mechanism
A mechanism in reinforcement learning that optimizes both embedding and reasoning quality.
Used in the second stage of the CoFree framework to ensure fine-grained relevance of reasoning.
nDCG@10
A metric for evaluating the performance of information retrieval systems, considering the relevance and ranking of results.
Used to evaluate CoFree's performance across multiple datasets.
Information Retrieval
The process of retrieving relevant information from large datasets.
CoFree aims to enhance the accuracy and efficiency of information retrieval systems.
Open Questions Unanswered questions from this research
- 1 How can CoFree be applied to larger datasets?
- 2 How to reduce computational costs to improve CoFree's accessibility?
Applications
Immediate Applications
Search Engine Optimization
Enhance search result relevance and accuracy by improving text embedding quality.
Long-term Vision
Intelligent Recommendation Systems
Apply CoFree in recommendation systems to improve personalized recommendation accuracy and user satisfaction.
Abstract
Large Language Models (LLMs) have recently shown strong potential for producing context-rich text embeddings for retrieval. Most existing methods either treat embedding learning as passive feature extraction or exploit LLM reasoning through instruction following for better embedding optimization. However, specialization toward embedding objectives can suppress useful reasoning generation or produce retrieval-irrelevant text. We refer to these two forms of degradation as reasoning collapse. To address this issue, we propose CoFree (Collapse-Free Reasoning Embedding), a two-stage framework that progressively integrates LLM reasoning into query and document embedding optimization while preserving reasoning quality. At the first stage, CoFree applies reference-guided supervised fine-tuning to restore the reasoning ability and retain representational strength of the foundation embedding model. At the second stage, we introduce dual rewards, an embedding-oriented reward and a reasoning-oriented reward, to guarantee fine-grained reasoning of the relevance toward the embedding goal in reinforcement learning. This endpoint-coupled optimization transforms embedding learning from static alignment into a high-quality reasoning-guided search process for retrieval. Extensive experiments demonstrate the effectiveness of CoFree, with CoFree-4B achieving an average absolute improvement of 2.8 nDCG@10 points over Qwen3-Embedding-4B across 22 datasets from MTEB and BRIGHT. Online experiments in a real-world retrieval system further show consistent gains. Code, RTED, and model checkpoints will be made publicly available.