Task-Adaptive Embedding Refinement via Test-time LLM Guidance

Embedding models have been widely used in information retrieval and classification tasks, primarily due to their ability to compute dense semantic representations for efficient online ranking. However, traditional embedding models often fall short in zero-shot classification tasks due to their lack of flexibility and instruction-following capabilities, which are strengths of generative large language models (LLMs). Recent advancements in generative LLMs have demonstrated remarkable performance in zero-shot tasks, yet their high computational cost makes them challenging to apply at corpus scale. Consequently, combining the efficiency of embedding models with the flexibility of generative LLMs has become a critical research direction.

Traditional embedding models are limited in zero-shot classification tasks due to their inflexibility and lack of instruction-following capabilities. Specifically, embedding models struggle to adapt to task-specific constraints when faced with ad-hoc user inputs, resulting in suboptimal performance in large-scale corpora compared to generative LLMs. However, the high computational cost of generative LLMs makes them impractical for large-scale applications. Therefore, improving embedding model performance in zero-shot tasks without increasing computational overhead remains a pressing challenge.

The core innovation of this paper is the introduction of a method that combines generative LLM feedback with embedding optimization. Specifically, the method leverages feedback from generative LLMs at test time to dynamically optimize user query embeddings, allowing them to adapt to task-specific constraints and improve performance in retrieval and classification tasks. Unlike traditional embedding models, this approach provides a new paradigm for task-adaptive embedding optimization by combining the flexibility of generative LLMs with the efficiency of embedding models.

• �� Generate initial document ranking using an embedding model.
• �� Obtain feedback scores from a generative LLM on a small set of documents.
• �� Use feedback scores to guide the embedding model in query optimization.
• �� Calculate similarity scores between the optimized query and documents.
• �� Update document ranking to enhance retrieval and classification task performance.

The experimental design includes testing multiple leading embedding models across tasks such as literature search, intent detection, and key-point matching. Datasets used include the arXiv computer science paper dataset, intent detection datasets, and key-point matching datasets. The experiments compare query performance before and after optimization, using evaluation metrics such as mean average precision (MAP) and recall. Additionally, ablation studies were conducted to verify the contribution of each component to overall performance.

Experimental results show that LLM-guided query refinement improved mean average precision by 16.9% in literature search, 9.4% in intent detection, and 15% in key-point matching. These results indicate that the proposed method effectively enhances embedding model performance across diverse tasks. Additionally, the experiments demonstrate that optimized queries better reflect task-specific constraints, improving the quality of document ranking.

The proposed method has potential value in multiple practical applications. For instance, in large-scale literature search, it significantly enhances retrieval accuracy and efficiency without increasing computational overhead, making it suitable for researchers and academic institutions. In customer intent analysis, optimized queries more accurately identify user intents, improving customer service quality. In key-point matching tasks, the method effectively enhances text matching precision, supporting large-scale opinion analysis.

Despite significant performance improvements across multiple tasks, the method's effectiveness is partly dependent on the quality of the generative LLM feedback, which may encode systematic biases or errors into the refined query representations. Additionally, in scenarios with extreme class imbalance, the initial retrieval set may overlook more informative documents, affecting optimization outcomes. Future research directions include exploring more sophisticated methods for constructing the feedback set, optimizing the selection of generative LLMs, and applying the method to other modalities.