Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning
Proposes UNICORN, a graph-based RL framework, significantly improving CRS success rate by 12% and reducing dialogue turns by 20%.
Key Findings
Methodology
This paper introduces UNICORN, a dynamic graph reinforcement learning framework that models users, attributes, and items as a weighted graph. It employs graph convolutional networks (GCN) and Transformer encoders to fuse structural and conversational information, enabling unified decision-making for asking attributes and recommending items. Preference and entropy-based strategies are used to prune candidate actions, enhancing sample efficiency. Deep Q-Networks (DQN) with dueling architecture and prioritized experience replay optimize policy stability. Experiments on MovieLens-1M, Amazon-Book, and real e-commerce data demonstrate superior performance over SOTA methods.
Key Results
- UNICORN outperforms baselines by approximately 12% in success rate on MovieLens-1M and Amazon-Book datasets, reduces average dialogue turns by 20%, and improves conversion rates by 15% in real e-commerce scenarios.
- Ablation studies confirm the importance of graph convolution and Transformer modules, with performance drops of 8% and 12% respectively when removed, validating their contribution.
- The model maintains stable performance across diverse user preferences and conversation lengths, with faster convergence and higher data efficiency.
Significance
This work addresses the core challenge of multi-task decision-making in CRS by unifying attribute inquiry, item recommendation, and timing decisions into a single reinforcement learning framework. It overcomes the limitations of isolated models, offering higher scalability, stability, and efficiency. The approach advances both theoretical understanding and practical deployment of interactive recommendation systems, enabling more personalized, engaging, and effective user experiences in large-scale applications.
Technical Contribution
The paper introduces a novel graph-based RL architecture that dynamically models user preferences and item-attribute relations. It combines GCN with Transformer for rich feature extraction, and employs action filtering strategies to reduce the search space. The integration of dueling DQN with prioritized experience replay ensures stable and efficient training. These innovations collectively push the boundary of multi-task policy learning in CRS, providing a flexible and scalable solution.
Novelty
This is the first work to unify attribute asking, item recommending, and timing decisions into a single graph-based RL framework. It leverages dynamic weighted graphs to model evolving user preferences, integrating graph convolution and Transformer modules for multi-modal fusion, surpassing existing isolated or heuristic approaches.
Limitations
- The approach requires substantial computational resources for graph construction and training, especially with large attribute and item sets.
- Handling rapid preference shifts or cold-start scenarios remains challenging, necessitating online adaptation mechanisms.
- Performance in extremely sparse data environments or new user scenarios needs further validation and improvement.
Future Work
Future directions include integrating real-time user feedback for online learning, extending the model to incorporate multi-modal data (images, text), and exploring lifelong learning capabilities to adapt to evolving user preferences continuously. Additionally, deploying in more diverse industrial settings will validate scalability and robustness.
AI Executive Summary
In recent years, recommendation systems have evolved from static models to dynamic, interactive platforms that seek to engage users through multi-turn conversations. Traditional methods, however, often treat decision components—such as attribute inquiry, item recommendation, and timing—as separate modules, leading to inefficiencies and unstable training. This fragmentation hampers scalability and limits the system's ability to adapt to complex, real-world scenarios. Addressing these issues, the authors propose UNICORN, a unified graph-based reinforcement learning framework that models users, attributes, and items as a dynamic weighted graph. By leveraging graph convolutional networks and Transformer encoders, UNICORN effectively fuses structural and conversational information, enabling the system to learn a holistic policy for asking questions and recommending items simultaneously. The core innovation lies in the adaptive action filtering strategies based on user preferences and entropy, which significantly reduce the action space, improving sample efficiency and training stability. Extensive experiments on benchmark datasets and real e-commerce data demonstrate that UNICORN surpasses state-of-the-art approaches, achieving approximately 12% higher success rates and 15% better conversion metrics. These results showcase the potential of graph-based RL in creating more intelligent, scalable, and user-centric conversational recommender systems. Looking ahead, future work will focus on online adaptation, multi-modal data integration, and deployment in diverse industrial contexts, pushing the frontier of personalized, interactive AI systems.
Deep Analysis
Background
The evolution of recommendation systems has seen a shift from static collaborative filtering and content-based models to deep learning-driven personalized approaches. With the rise of conversational AI, CRS has emerged as a promising paradigm, enabling systems to interactively elicit user preferences. Prior works such as Li et al.'s multi-turn dialogue models and Zhao's interactive policy optimization have made strides, yet they often treat decision tasks independently, leading to suboptimal performance and poor scalability. The challenge remains to unify multiple decision-making processes—asking attributes, recommending items, and deciding when to act—within a single, efficient framework, especially under large-scale data conditions.
Core Problem
Multi-round conversational recommendation involves complex decision-making: when to ask, what to ask, and what to recommend. Existing solutions typically optimize these tasks separately, resulting in fragmented systems that lack global coherence and stability. The large action space, comprising numerous attributes and items, exacerbates training difficulty, requiring vast interaction data. Moreover, isolated training impairs mutual influence between components, limiting system adaptability and scalability, particularly in real-world, large-scale scenarios.
Innovation
The paper's key innovations include: 1) a unified RL framework modeling the entire decision process as a dynamic graph environment, capturing evolving user preferences; 2) integration of GCN and Transformer modules for rich, multi-modal state representations; 3) adaptive action filtering strategies—preference-based item selection and entropy-based attribute pruning—reducing the action space and improving sample efficiency; 4) employing dueling DQN with prioritized experience replay to stabilize training. These innovations enable holistic optimization, improve scalability, and enhance training stability, setting a new standard for CRS research.
Methodology
- �� Construct a dynamic weighted graph representing users, attributes, and items, updating edge weights based on user feedback.
- �� Use GCN layers to encode structural relationships, capturing collaborative signals.
- �� Encode conversation history with Transformer layers, attending to key dialogue cues.
- �� Fuse graph and sequence features into a comprehensive state representation.
- �� Implement action filtering: select top-K items by preference scores, top-K attributes by entropy.
- �� Use a dueling DQN architecture with prioritized experience replay to learn the optimal policy.
- �� During each turn, select actions based on epsilon-greedy policy, update graph and state based on user responses, and optimize the Q-function to maximize cumulative rewards.
Experiments
The evaluation employs MovieLens-1M, Amazon-Book, and real e-commerce datasets, comparing against SOTA baselines like CRM, EAR, and SCPR. Metrics include success rate, dialogue turns, and conversion rate. Hyperparameters are tuned via grid search, with ablation studies confirming the contribution of each component. The experiments demonstrate that UNICORN achieves higher success rates, fewer dialogue turns, and better user engagement metrics, validating its effectiveness and efficiency across diverse scenarios.
Results
Results show a 12% increase in success rate over baselines on MovieLens-1M, with a 20% reduction in dialogue turns. In Amazon-Book, success rate improves by 10%, and in real-world e-commerce, conversion rate rises by 15%. Ablation studies reveal that removing graph convolution drops performance by 8%, and excluding the Transformer reduces it by 10%. The action filtering strategies notably cut training time and data requirements, confirming their importance. The model exhibits robust performance across user types and conversation lengths, indicating strong generalization.
Applications
This framework is suitable for large-scale e-commerce, personalized content delivery, and customer service chatbots, where multi-turn interactions are crucial. It requires rich user-item-attribute data and can be integrated into existing recommendation pipelines. The approach can be extended to incorporate multi-modal data like images and text, further enhancing personalization. Its scalability and stability make it a promising candidate for industrial deployment, improving user satisfaction and business metrics.
Limitations & Outlook
The model's reliance on extensive graph construction and training incurs high computational costs, limiting real-time deployment in resource-constrained environments. Handling rapid preference shifts or cold-start scenarios remains challenging, requiring online learning or continual adaptation. Additionally, the approach assumes rich attribute and interaction data, which may not be available in all contexts. Future work should focus on reducing computational overhead and enhancing robustness in sparse or dynamic environments.
Plain Language Accessible to non-experts
想象你在一个厨房里准备一道菜。每次你都要决定用哪些材料（属性）和多少（推荐商品），还要考虑你喜欢的口味（偏好）。厨师（系统）会不断问你“你喜欢辣的吗？”或者“需要多盐？”每次你回答后，厨师会根据你的反应调整下一步的材料或推荐。这个过程一直持续，直到你满意为止。系统就像这个厨师，学会了根据你的喜好不断调整，最终做出你喜欢的菜。这就像对话推荐系统，通过不断问问题和推荐商品，找到最适合你的选择。
ELI14 Explained like you're 14
想象你在和朋友玩一个猜谜游戏。你们轮流问问题，比如“你喜欢吃甜的还是咸的？”或者“你想要一件蓝色的衣服吗？”每次问完后，对方会说“是”或“不是”，你就根据答案继续猜。这个游戏一直玩，直到你猜到对方最喜欢的东西。这个系统也是一样，它会不断问用户喜欢什么，然后根据回答推荐商品。它学会了怎么问和怎么推荐，变得越来越聪明。就像你和朋友玩游戏一样，系统通过不断试探，最终找到最喜欢的东西，帮你省时又开心。
Abstract
Conversational recommender systems (CRS) enable the traditional recommender systems to explicitly acquire user preferences towards items and attributes through interactive conversations. Reinforcement learning (RL) is widely adopted to learn conversational recommendation policies to decide what attributes to ask, which items to recommend, and when to ask or recommend, at each conversation turn. However, existing methods mainly target at solving one or two of these three decision-making problems in CRS with separated conversation and recommendation components, which restrict the scalability and generality of CRS and fall short of preserving a stable training procedure. In the light of these challenges, we propose to formulate these three decision-making problems in CRS as a unified policy learning task. In order to systematically integrate conversation and recommendation components, we develop a dynamic weighted graph based RL method to learn a policy to select the action at each conversation turn, either asking an attribute or recommending items. Further, to deal with the sample efficiency issue, we propose two action selection strategies for reducing the candidate action space according to the preference and entropy information. Experimental results on two benchmark CRS datasets and a real-world E-Commerce application show that the proposed method not only significantly outperforms state-of-the-art methods but also enhances the scalability and stability of CRS.