Customer-R1: Personalized Simulation of Human Behaviors via RL-based LLM Agent in Online Shopping
Customer-R1 employs RL with explicit user personas, boosting next-action prediction accuracy from 7.32% to 39.58%, outperforming baselines.
Key Findings
Methodology
Customer-R1 integrates explicit user personas into a reinforcement learning framework utilizing the GRPO algorithm. The model inputs include user history, web observation, and persona, generating rationales and actions. Rewards are designed based on action correctness and format adherence, guiding the model to produce semantically coherent, personalized behaviors. The backbone is Qwen2.5-7B, fine-tuned via supervised learning (SFT) before RL optimization. Experiments on the OPeRA dataset demonstrate significant improvements in next action accuracy, from 7.32% (zero-shot) to 39.58%, surpassing prompt and SFT baselines. The approach emphasizes behavioral fidelity and diversity, validated through ablation studies showing the importance of persona conditioning.
Key Results
- Customer-R1 achieves a next action prediction accuracy of 39.58%, a notable increase over baseline models, with behavior distributions closely matching user personas, indicating high personalization fidelity.
- Inclusion of explicit user personas and rationales significantly improves performance, while shuffling or omitting these signals degrades results, confirming their critical role.
- Longer context windows (65k tokens) enhance stability and accuracy, especially in complex behavior scenarios, with larger models (7B parameters) outperforming smaller ones (3B).
Significance
This work advances personalized human behavior modeling by integrating explicit user profiles into RL-based simulation, addressing the limitations of generic models. It provides a scalable, verifiable framework for realistic user simulation, with implications for personalized recommendation, user experience enhancement, and behavioral analytics. The method bridges the gap between average-user models and individual-specific simulations, paving the way for more nuanced understanding and prediction of user actions in online environments. Its potential to improve recommendation accuracy and user engagement marks a significant step forward in intelligent system design.
Technical Contribution
The core innovation lies in conditioning RL policies on explicit user personas, combined with a verifiable reward structure that enforces semantic and format correctness. The use of GRPO ensures stable policy optimization, while the prompt engineering effectively incorporates persona information. This approach differs from prior work by explicitly modeling individual differences and optimizing for behavior fidelity, thus enabling more personalized and realistic simulations. The framework also demonstrates how to leverage synthetic rationales to improve reasoning and action quality, contributing to the broader field of behavior modeling.
Novelty
This study is the first to systematically embed explicit user personas into a reinforcement learning framework for step-wise behavior simulation, validated on real-world datasets. Unlike prior methods that rely solely on prompt engineering or supervised fine-tuning, it combines these with reward-guided policy refinement, achieving higher fidelity and personalization. The integration of verifiable rewards and explicit persona conditioning represents a significant innovation, setting a new standard for personalized behavior modeling in online environments.
Limitations
- Dependence on high-quality, accurate user personas limits robustness; inaccuracies or missing data can impair simulation fidelity.
- RL training incurs high computational costs, especially with large models and long contexts, restricting scalability.
- Data limitations restrict behavior diversity, making it challenging to generalize to unseen or rare user behaviors. Future work should explore multi-modal data integration and more efficient training strategies.
Future Work
Future research will focus on incorporating multi-modal user data, such as images and voice, to enrich behavior modeling. Efforts will also aim to reduce training costs through more efficient algorithms and transfer learning techniques. Additionally, expanding datasets to cover broader user scenarios and behaviors will improve model robustness. Exploring multi-objective RL to balance behavior diversity and fidelity, as well as deploying these models in real-world recommendation systems, represents promising directions.
AI Executive Summary
In the rapidly evolving field of human behavior simulation, the ability to generate personalized, step-wise user actions remains a significant challenge. Traditional methods, primarily based on prompt engineering or supervised fine-tuning, often produce generic behaviors that fail to capture individual differences. This limitation hampers the development of truly personalized online experiences, especially in e-commerce and recommendation systems. Addressing this gap, the present study introduces Customer-R1, a reinforcement learning framework that explicitly conditions behavior generation on user personas. By integrating detailed user profiles and employing the Group Relative Policy Optimization (GRPO) algorithm, the model learns to produce behavior sequences that are both accurate and aligned with individual preferences.
The core innovation lies in the reward design, which combines action correctness and format adherence, ensuring behaviors are semantically coherent and personalized. The model is built upon the Qwen2.5-7B architecture, first fine-tuned via supervised learning, then refined through RL. Extensive experiments on the OPeRA dataset demonstrate that Customer-R1 significantly outperforms baseline methods, with next action prediction accuracy rising from 7.32% to 39.58%. The results also show that incorporating explicit user personas and rationales enhances behavioral fidelity, making the simulation more realistic and user-specific.
This work has profound implications for online shopping, personalized recommendation, and behavioral analytics. It offers a scalable, verifiable approach to modeling individual user behaviors, bridging the gap between generic and personalized simulations. The findings suggest that future systems can leverage such models to deliver more engaging, tailored experiences, ultimately transforming how digital platforms understand and serve their users.
Despite these advances, challenges remain. The reliance on high-quality user data, the computational costs of RL training, and the limited diversity of current datasets pose hurdles for widespread deployment. Future research will focus on multi-modal data integration, more efficient training algorithms, and broader data collection to enhance model robustness. Overall, Customer-R1 marks a significant step toward truly personalized human behavior modeling in AI, promising richer, more human-like interactions in digital environments.
Deep Analysis
Background
人类行为模拟在心理学、社会科学和人机交互领域具有悠久的研究传统。早期研究多关注宏观行为描述，近年来随着大规模语言模型（LLMs）的崛起，逐步转向细粒度、step-wise的行为预测。代表性工作如Lu et al.（2025a）利用监督微调（SFT）在私有数据上训练模型，提升行为预测准确率。Zhang et al.（2025b）提出Shop-R1，通过强化学习（RL）优化行为生成，显著改善了行为的合理性和多样性。然而，这些方法多关注平均用户行为，缺乏对个体差异的建模，限制了实际应用效果。近年来，用户画像逐渐被引入行为模拟中，但多停留在离线分析或未结合强化学习优化，缺少系统性框架。本文在此基础上，结合强化学习和显式画像信息，提出了更具个性化和真实性的行为模拟方案。
Core Problem
核心问题在于如何在step-wise行为模拟中融入个体差异，实现高保真度的个性化仿真。现有方法多忽视用户画像的作用，导致生成行为缺乏个性化特征，难以反映不同用户的偏好和目标。同时，强化学习虽能优化策略，但奖励设计缺乏有效性，导致行为偏离真实用户。解决这一问题对于提升推荐系统的用户体验、理解用户行为模式具有重要意义，但面临数据稀缺、奖励设计复杂和模型泛化能力不足等挑战。
Innovation
本研究的创新点包括：1）引入显式用户画像作为条件信息，增强模型对个体差异的感知；2）设计可验证的奖励机制，结合动作正确性和格式一致性，确保行为合理性；3）采用GRPO强化学习算法，优化step-wise行为生成策略，兼顾模型稳定性和性能；4）结合prompt工程，增强模型对画像信息的利用能力。这些创新突破了以往只依赖prompt或SFT的局限，使行为模拟更具个性化和真实性。
Methodology
- �� 输入：用户历史行为、网页状态、用户画像。
- �� 模型结构：基于Qwen2.5-7B，结合prompt引导，输出行为 rationale 和动作预测。
- �� 训练策略：先用SFT微调，确保基础行为理解，再用GRPO强化学习优化行为策略。
- �� 奖励设计：动作正确性奖励（完全匹配得1分），格式奖励（符合JSON schema得1分），加权提升复杂行为的奖励。
- �� 训练过程：多轮采样、奖励回传、策略更新，确保模型学习到个性化行为规律。
Experiments
采用OPeRA数据集，包含527个真实购物会话，行为总数5856，涵盖多样行为类型。模型在不同设置（零样本、SFT、RL、SFT+RL）下进行训练和评估。指标包括下一动作准确率、行为类型F1、细粒度预测准确率和会话终止预测。通过对比不同模型规模和上下文长度，验证模型的鲁棒性和效果。采用synthetic rationale增强，提升模型理解能力。实验结果显示，SFT+RL模型在所有指标上表现最佳，准确率达39.58%，比基线提升显著。
Results
模型在下一动作预测中的准确率由零样本的7.32%提升至39.58%，在行为类型和细粒度预测上也表现优异。引入用户画像和 rationale 后，模型行为更贴合用户偏好，表现出更强的个性化能力。长上下文（65k tokens）进一步提升模型稳定性和预测准确性。 Ablation研究验证了画像和 rationale 对性能的关键作用，去除任何一项都会显著降低效果。整体来看，模型在行为模拟的真实性和多样性方面取得了突破。
Applications
该方法可应用于个性化推荐、用户行为分析、虚拟用户生成等场景。通过模拟真实用户行为，提升推荐系统的准确性和用户体验。未来还可结合多模态信息（如图片、语音）实现更复杂的用户行为仿真，为智能客服、虚拟助手等提供支持。长远目标是实现全场景、多维度的用户行为建模，推动个性化服务的智能化升级。
Limitations & Outlook
模型高度依赖用户画像的准确性，若画像信息不完整或偏差，模拟效果会受影响。强化学习训练成本高，尤其在大模型和长上下文中，资源消耗大。此外，数据集有限，难以覆盖所有实际场景，未来需扩展多样化数据源，提升模型的泛化能力。模型在极端行为或新颖场景下表现仍有待优化。
Plain Language Accessible to non-experts
想象你在一家餐厅点餐，服务员会根据你的喜好和之前的点餐习惯推荐菜品。这个系统就像一个非常聪明的服务员，能记住你喜欢吃什么、偏好什么口味，还能根据你之前的选择预测你下一次可能会点什么。它不仅知道你喜欢辣的还是不辣的，还能根据你的偏好推荐不同的菜肴。为了做到这一点，它学习了很多人的点餐习惯，结合你的个人信息，模拟出你真实的用餐行为。这样一来，每次你点餐时，服务员都能给出符合你口味的建议，让你觉得像和朋友一样贴心。这种模拟不仅让餐厅的服务更智能，也让你体验到更个性化的服务。
ELI14 Explained like you're 14
想象你在学校里，有个超级聪明的朋友，他能记住你平时喜欢什么、做过什么，然后帮你预测下一次你会做什么。比如，你喜欢玩足球，他就会猜到你下一次可能会去操场踢球。这个朋友还会根据你的喜好，给你推荐一些你可能喜欢的东西，比如喜欢的游戏或者书。它学习了很多人的习惯，然后用这些信息，模拟出你真实的行为。这样，你每次做决定时，它都能帮你想好下一步，像个贴心的助手一样。这个系统让我们的生活变得更方便、更有趣，因为它懂得我们每个人的特别之处，就像有个专属的朋友一样。
Abstract
Simulating step-wise human behavior with Large Language Models (LLMs) has become an emerging research direction, enabling applications in various practical domains. While prior methods, including prompting, supervised fine-tuning (SFT), and reinforcement learning (RL), have shown promise in modeling step-wise behavior, they primarily learn a population-level policy without conditioning on a user's persona, yielding generic rather than personalized simulations. In this work, we pose a critical question: how can LLM agents better simulate personalized user behavior? We introduce Customer-R1, an RL-based method for personalized, step-wise user behavior simulation in online shopping environments. Our policy is conditioned on an explicit persona, and we optimize next-step rationale and action generation via action correctness reward signals. Experiments on the OPeRA dataset emonstrate that Customer-R1 not only significantly outperforms prompting and SFT-based baselines in next-action prediction tasks, but also better matches users' action distribution, indicating higher fidelity in personalized behavior simulation.