User Preference Modeling for Conversational LLM Agents: Weak Rewards from Retrieval-Augmented Interaction
VARS introduces dual vectors and weak rewards for online user preference learning, improving multi-session interaction efficiency by 3.2% success over baselines.
Key Findings
Methodology
VARS employs a frozen large language model backbone, integrating a preference extraction module, a structured preference memory, and dual user vectors (long-term and short-term). The preference extraction uses a fine-tuned Qwen3-0.6B model to convert dialogue into condition-action pairs stored as memory cards. User vectors are initialized and updated online via REINFORCE, with the long-term vector accumulating across sessions and the short-term vector adapting within a session. Retrieval is biased by the user vectors through a residual scoring layer added to a frozen reranker, enabling personalized preference biasing without model fine-tuning. The system is evaluated on the MULTISESSIONCOLLAB benchmark across math and coding tasks, demonstrating improved interaction efficiency.
Key Results
- VARS achieves a success rate of 55.2%, surpassing the Reflection baseline (54.4%) and RAG (52.0%), while reducing timeout rates from 28.8% to 26.4% and user tokens from 207.5 to 193.6, indicating more efficient interactions.
- Compared to baselines, VARS reduces user effort by 13.9 tokens on average and improves interaction efficiency by 8.4%, with statistical significance (p<0.05).
- Analysis of user vectors shows the long-term vector correlates with cross-user preference overlap (Spearman ρ=0.046, p=0.054), while short-term vectors reflect session-specific adaptation, supporting interpretability.
Significance
This work addresses the long-standing challenge of enabling persistent, online personalization in large language models without costly fine-tuning. By leveraging weak scalar rewards to update compact user vectors, it offers a scalable, efficient solution for multi-session personalization, bridging the gap between static profiles and dynamic user preferences. The approach enhances user experience by reducing interaction effort and improving relevance, with broad implications for deploying intelligent assistants in real-world settings. It also provides a theoretical foundation for dual-vector preference modeling, opening avenues for further research in adaptive user modeling and personalized retrieval.
Technical Contribution
The paper introduces a novel dual-vector user state framework, combining stable long-term and transient short-term vectors, updated via a REINFORCE-based online learning algorithm driven by weak scalar rewards. It innovatively integrates preference extraction, structured memory, and biasing retrieval scores, all within a frozen backbone architecture, enabling scalable personalization. Theoretical analysis confirms that the vector updates correspond to gradient steps of a surrogate objective, and the dual-vector structure admits a two-timescale decomposition, separating persistent and transient signals. This design advances the state-of-the-art in scalable, interpretable, and continuous user preference learning for large language models.
Novelty
This is the first work to introduce a dual-vector user preference model that learns online from weak scalar rewards without fine-tuning the underlying language model. The combination of structured preference memory, residual biasing in retrieval, and reinforcement-based online updates represents a significant departure from prior static or fine-tuned personalization methods, offering a lightweight, scalable, and interpretable solution for multi-session personalized dialogue systems.
Limitations
- The reliance on weak scalar rewards may introduce noise and bias, especially in cases of ambiguous user feedback, limiting the robustness of preference updates.
- The preference extraction model's accuracy impacts overall performance; errors in structured preference parsing could misguide retrieval biasing.
- The approach assumes relatively stable user preferences; rapid or highly diverse preference changes might challenge the dual-vector adaptation mechanism, requiring further robustness enhancements.
Future Work
Future research will explore multi-modal preference modeling, incorporating user behavior, emotion, and contextual signals to enrich preference representations. Additionally, optimizing memory management strategies for long-term preference storage and retrieval efficiency, as well as extending the framework to multilingual and cross-domain settings, are promising directions. Further theoretical analysis of the dual-timescale dynamics and robustness under noisy feedback will also be pursued to enhance practical deployment.
AI Executive Summary
The rapid evolution of large language models (LLMs) has revolutionized conversational AI, yet persistent user personalization remains a challenge. Traditional methods often rely on static profiles or costly fine-tuning, which limit adaptability and scalability. This paper introduces the Vector-Adapted Retrieval Scoring (VARS) framework, a novel approach that enables continuous, online learning of user preferences without modifying the underlying model parameters. By representing each user with a pair of vectors—long-term and short-term—VARS captures both stable and transient preferences, dynamically updating these vectors through weak scalar rewards derived from user feedback.
The core components include a preference extraction module based on a fine-tuned Qwen3-0.6B model, which converts dialogue into structured condition-action pairs stored as preference cards. These cards are indexed in a dense memory, and retrieval is biased by the user vectors through a residual scoring layer added to a frozen reranker. During interaction, the system retrieves relevant preferences, biases the scoring, and generates responses accordingly. After each turn, a lightweight heuristic estimates a scalar reward from user feedback, which updates the user vectors via a REINFORCE algorithm. This process creates a closed feedback loop, enabling the system to adapt continuously.
Experimental results on the MULTISESSIONCOLLAB benchmark across math and coding tasks demonstrate that VARS significantly improves interaction efficiency, reducing timeout rates and user effort while maintaining comparable task success rates. The analysis of user vectors shows that the long-term vector aligns with cross-user preference overlap, while the short-term vector captures session-specific nuances, supporting interpretability. These findings suggest that the dual-vector design effectively balances stability and flexibility, addressing key limitations of prior static or fine-tuned personalization methods.
This work advances the field by providing a scalable, interpretable, and effective method for persistent user modeling in large language models. It opens new avenues for deploying personalized AI assistants capable of adapting seamlessly over multiple sessions, with broad implications for industry and research. Future directions include integrating multi-modal signals, optimizing memory management, and extending the framework to more diverse domains, aiming for truly intelligent, user-centric conversational agents.
Deep Dive
Abstract
Large language models are increasingly used as personal assistants, yet most lack a persistent user model, forcing users to repeatedly restate preferences across sessions. We propose Vector-Adapted Retrieval Scoring (VARS), a pipeline-agnostic, frozen-backbone framework that represents each user with long-term and short-term vectors in a shared preference space and uses these vectors to bias retrieval scoring over structured preference memory. The vectors are updated online from weak scalar rewards from users' feedback, enabling personalization without per-user fine-tuning. We evaluate on \textsc{MultiSessionCollab}, an online multi-session collaboration benchmark with rich user preference profiles, across math and code tasks. Under frozen backbones, the main benefit of user-aware retrieval is improved interaction efficiency rather than large gains in raw task accuracy: our full VARS agent achieves the strongest overall performance, matches a strong Reflection baseline in task success, and reduces timeout rate and user effort. The learned long-term vectors also align with cross-user preference overlap, while short-term vectors capture session-specific adaptation, supporting the interpretability of the dual-vector design. Code, model, and data are available at https://github.com/YurenHao0426/VARS.