SCoRD: Semantic-Assisted Continual Retriever-Reranker Distillation for LLM-Based Recommendation

The evolution of recommendation systems has transitioned from traditional collaborative filtering methods like matrix factorization to deep learning models such as SASRec and BERT4Rec, which leverage sequential and semantic information. Recently, large language models (LLMs) like GPT and PaLM have demonstrated exceptional semantic reasoning capabilities, enabling more nuanced user preference modeling. However, these models are computationally expensive, limiting their deployment in real-time systems. Existing solutions include retrieve-then-rank pipelines, where lightweight retrievers filter candidates and LLMs re-rank them, but static training cannot keep pace with dynamic user interests. Knowledge distillation (KD) has been used to transfer the richness of LLMs into smaller models, but most methods focus on static settings, lacking mechanisms for continual adaptation in streaming data environments. The challenge remains to design an efficient, scalable framework that allows ongoing model updates without incurring prohibitive costs.

The core problem addressed is how to enable continual co-adaptation of a lightweight retriever and a costly LLM-based reranker in a non-stationary data stream. Frequent updates of the reranker are computationally prohibitive, yet infrequent updates cause the guidance to become stale, impairing the retriever’s ability to reflect recent trends. Conversely, updating only the retriever limits its capacity to incorporate semantic understanding from the LLM. Existing methods like static distillation or periodic CCD do not adequately balance these trade-offs, leading to suboptimal performance in dynamic environments. The key challenge is to develop a mechanism that preserves up-to-date semantic guidance for the retriever, reduces LLM inference costs, and maintains high recommendation quality amid evolving user interests.

The main innovations include: 1) a semantic reasoning assistant that distills LLM’s inference of user intents into a compact, reusable memory, avoiding frequent LLM calls; 2) a selective distillation strategy that targets low-confidence sequences, reducing computational overhead; 3) an intent drift signal that captures evolving user interests and guides the retriever’s continuous updates; 4) a multi-stage training process that aligns the retriever, assistant, and reranker, ensuring mutual adaptation. These innovations collectively address the high cost and limited capacity issues of existing methods, enabling efficient, real-time continual learning in recommendation pipelines.

• �� Construct a semantic reasoning assistant by organizing LLM-inferred intents into a dynamic memory on the base data block.
• �� During LLM-to-retriever distillation, select sequences with low confidence, and use the assistant to extract intents, which are then distilled into the retriever.
• �� Expand the intent memory with verified intents via a verification process that involves held-out item prediction.
• �� Enable the retriever to update frequently using the latest behavioral signals and inferred intents, without invoking LLM.
• �� Feed retriever representations and intent drift signals back into the reranker to align semantic reasoning.
• �� Jointly train the assistant and retriever with objectives for reasoning distillation and semantic recommendation, ensuring mutual enhancement.
• �� During reranker updates, incorporate retriever’s latest knowledge to refine semantic understanding, closing the loop.

Using datasets from e-commerce, content platforms, and social media, the authors compare SCoRD against static distillation and CCD baselines. Metrics include Recall@N, NDCG, and computational costs. They simulate dynamic interest scenarios with rapid interest shifts and new user/item arrivals. Hyperparameters such as distillation frequency, intent confidence thresholds, and drift signal weights are tuned. Ablation studies assess the impact of each component, especially the intent memory and selective distillation. Results demonstrate that SCoRD achieves superior recommendation accuracy, faster adaptation, and lower inference costs, validating its effectiveness in real-world streaming environments.

SCoRD outperforms baselines with over 15% higher Recall@10 and 10% higher NDCG across datasets. Cost analysis shows 40% reduction in LLM inference, while adaptation speed improves by 30%. Ablation confirms that intent memory and selective distillation are critical for performance gains. The framework maintains robustness under various interest change rates, demonstrating its scalability and practical value.

Applicable in online retail, media streaming, and social platforms, SCoRD enables real-time personalized recommendations that adapt to user behavior shifts. It requires only initial intent memory training, after which continual updates are efficient. Its design suits large-scale deployment, reducing operational costs while enhancing user engagement. Future integration with multi-modal data and federated learning can further expand its industry impact.

Dependence on accurate intent inference may limit performance if user behaviors are highly ambiguous. In scenarios with extremely rapid or unpredictable interest shifts, the intent memory may lag, reducing effectiveness. Computational overhead for large-scale deployment still exists, requiring further optimization. Future work should focus on improving intent extraction robustness, scalability, and multi-modal integration to address these challenges.