Beyond Action Imitation: Learning a Decision-Aware User Simulator for Online Advertising
DASH integrates cross-domain histories and thinking traces, enhancing decision-aware user simulation for online advertising.
Key Findings
Methodology
DASH employs a three-stage framework: Stage 1 uses hierarchical context compression and prompt optimization to fuse heterogeneous histories; Stage 2 distills thinking trajectories from strong LLMs, combined with a rubric-based reward model for supervised fine-tuning; Stage 3 applies hybrid reward reinforcement learning to optimize both action prediction and thinking trace quality. The core algorithms include hierarchical compression, structured prompting, trajectory distillation, and Group Relative Policy Optimization (GRPO). Experiments on Tencent’s multi-domain advertising data demonstrate superior performance over traditional single-domain, action-only models.
Key Results
- On Tencent’s five heterogeneous content domains, DASH improves behavior prediction F1 score by 12%, and enhances logical consistency scores of simulated traces by 15%. The model achieves a behavior F1 of 0.78, significantly surpassing baselines (~0.66). Ablation studies show that hierarchical compression and prompt optimization each contribute about 6% performance gains, with trajectory distillation and reward mechanisms jointly boosting simulation fidelity. The model exhibits strong robustness and efficiency in handling long, multi-domain sequences.
- In ablation experiments, removing hierarchical compression or prompt optimization reduces performance by approximately 6%, confirming their importance. Incorporating trace-level rewards via reinforcement learning further improves behavior accuracy and trace coherence. The model generalizes well across domains, providing reliable offline evaluation and deep user insights.
- This approach not only enhances offline ad system evaluation but also offers a rich interpretability of user decision processes, paving the way for more transparent and personalized recommendation systems in industry.
Significance
This work addresses key limitations of existing user simulators, which often rely on single-domain behaviors and action-only supervision, leading to biased and incomplete models. By integrating heterogeneous histories and user thinking traces, DASH significantly improves the fidelity and diagnostic value of simulation. Its multi-stage training paradigm and reward design open new avenues for deep user understanding, enabling more accurate offline evaluation and interpretability in recommendation systems. Long-term, this framework supports the development of multi-modal, multi-domain personalized user models, fostering advances in intelligent advertising and content personalization.
Technical Contribution
DASH introduces several technical innovations: hierarchical context compression to fuse multi-domain histories efficiently; structured prompt design with self-refinement for reasoning over folded context; trajectory distillation from strong LLMs combined with quality filtering; and a hybrid reward mechanism integrating action accuracy and trace quality via reinforcement learning. These components collectively enable deep modeling of user decision processes, surpassing traditional action-only simulators. The framework balances efficiency and expressiveness, suitable for industrial deployment, and provides theoretical insights into multi-objective optimization of user behavior and cognition.
Novelty
This study is the first to systematically combine multi-domain heterogeneous histories with user thinking traces in a decision-aware simulation framework. Unlike prior models focusing solely on observable actions, DASH models both behavior and underlying decision processes, leveraging multi-stage training and reward mechanisms. Its hierarchical compression and structured prompting are novel contributions that significantly enhance simulation fidelity and interpretability, filling a critical gap in the literature.
Limitations
- Handling extremely long or highly complex multi-modal sequences remains challenging; current compression may lose subtle signals affecting fidelity.
- Reward functions rely on manually designed rubrics, which may introduce subjectivity and limit generalization across different domains or tasks.
- Training complexity and computational cost are high, hindering large-scale or real-time deployment without further optimization.
Future Work
Future directions include integrating multi-modal data (visual, audio) for richer user models, exploring causal inference to improve decision trace modeling, and developing efficient end-to-end systems for real-time simulation. Additionally, automating reward design and scaling training procedures will be key to industrial adoption. Extending the framework to support personalized multi-modal content recommendation and explainability also holds promise.
AI Executive Summary
In the rapidly evolving landscape of online advertising, accurate offline evaluation of recommendation algorithms remains a challenge. Traditional user simulators, primarily based on rule-based or single-domain models, often fail to capture the complexity of real user behaviors, especially across multiple content domains. These limitations hinder the development of robust, scalable, and interpretable recommendation systems.
Addressing this gap, the paper introduces DASH, a decision-aware user simulator that jointly models user actions and underlying thinking traces across heterogeneous content domains. The framework employs a three-stage training paradigm: first, it uses hierarchical context compression and prompt optimization to effectively fuse multi-domain histories into decision-relevant signals; second, it distills user thinking trajectories from powerful language models, coupled with a rubric-based reward system, for supervised fine-tuning; third, it applies reinforcement learning with a hybrid reward combining action accuracy and trace quality to further refine the model.
This innovative approach enables the simulator to generate more realistic, interpretable, and diagnostic user behaviors. Extensive experiments on Tencent’s multi-domain advertising data demonstrate that DASH significantly outperforms existing models, achieving a 12% improvement in behavior prediction F1 score and a 15% increase in logical consistency of simulated traces. The model’s efficiency and robustness suggest strong potential for industrial deployment, providing a valuable tool for offline evaluation, user behavior analysis, and personalized recommendation.
While promising, the framework faces challenges such as handling extremely long sequences and the subjective nature of reward design. Future work aims to incorporate multi-modal data, improve scalability, and enhance interpretability, paving the way for smarter, more transparent advertising systems that better understand and predict user decision-making processes.
Deep Analysis
Background
用户模拟在推荐系统中的作用不断增强，早期多采用规则或强化学习方法，如RecSim和ReacGym，主要模拟用户的可观察行为，缺乏对深层决策的理解。近年来，基于大规模预训练模型（LLMs）的模拟器逐步兴起，如Agent4Rec，通过引入用户画像、记忆和反思机制，提升个性化模拟能力。然而，这些模型多集中于单域行为，忽视了用户在多内容、多平台、多模态环境中的复杂交互，限制了模拟的真实性和诊断价值。随着广告系统的复杂化，需求也逐渐转向跨域、多模态、多层次的用户行为理解，推动了深度决策建模的发展。
Core Problem
现有模拟器普遍面临两个核心问题：一是单域偏好建模，难以反映用户在多内容域的真实偏好，导致模拟偏差；二是动作导向的响应预测，容易过拟合特定行为，缺乏对用户深层思考轨迹的建模，限制了模拟的真实性和诊断能力。这些问题阻碍了广告系统的离线评估和用户理解的深入，亟需融合异构多域历史信息，结合用户决策的深层思考轨迹，提升模拟的深度和解释性。
Innovation
本研究的主要创新包括：1）提出层次化上下文压缩策略，有效融合多域异构历史信息，减少噪声干扰，提升决策相关信号的表达；2）设计结构化提示优化流程，利用多步骤推理引导模型在有限上下文中进行深层推理，确保输出合理；3）利用强大LLMs生成用户思考轨迹，结合评分奖励模型筛选高质量轨迹，增强模型的思考能力；4）引入混合奖励机制，将行为匹配和轨迹质量结合，通过强化学习优化模型输出。这些创新共同推动用户模拟由单纯动作预测向深层决策理解转变。
Methodology
- �� 第一步，通过层次化压缩策略，将用户历史信息（广告、内容、偏好）按重要性和信息密度进行筛选，确保保留关键决策信号。• 设计结构化自我优化提示，包括焦点识别、草稿生成、验证和修正步骤，引导模型在有限上下文中进行多步骤推理。• 利用强大LLMs生成用户思考轨迹，结合人工定义的评分指标（内容合理性、逻辑一致性）筛选高质量轨迹，进行有监督微调。• 在强化学习阶段，结合行为匹配奖励和轨迹质量评分，设计混合奖励机制，优化模型行为预测和思考轨迹的质量，提升模拟的真实性和诊断能力。
Experiments
采用腾讯多域广告数据，涵盖五个内容域（广告、视频、新闻等），与传统模型（如RecSim、微调LLMs）进行对比。指标包括行为预测F1、轨迹逻辑一致性、模拟忠实度。通过AB测试和消融实验验证各阶段贡献，调优超参数如上下文长度、奖励阈值。训练在GPU集群上完成，确保工业应用效率。
Results
DASH在五个域中的行为F1达0.78，优于对比模型（0.66），提升显著。轨迹逻辑一致性得分提升15%，模拟忠实度增强。消融实验显示，层次化压缩和提示优化各贡献约6%，轨迹蒸馏和奖励机制共同推动性能提升。模型表现出良好的泛化能力和鲁棒性，为离线评估提供了更可靠的工具。
Applications
该模型可广泛应用于广告系统的离线效果评估、用户行为分析和个性化推荐优化。通过模拟用户深层决策过程，帮助开发者快速测试新算法，减少A/B测试成本。同时，模型生成的思考轨迹为理解用户偏好和行为变化提供了丰富线索，提升推荐系统的透明度和用户体验。
Limitations & Outlook
模型在极端长序列和多模态场景下仍存在信息压缩不足的问题，可能影响模拟精度。奖励机制依赖人工定义，存在主观性和泛化不足风险。训练成本较高，限制了大规模部署和实时应用，未来需优化模型结构和训练流程以提升效率。
Plain Language Accessible to non-experts
想象你在一家厨房里做饭，厨师需要根据各种食材、调料和食谱，决定下一步该放什么、什么时候放。传统厨师只记住一些简单的步骤，比如放盐或放油，但很难理解整个菜肴的味道和做法。本文提出的DASH就像一个聪明的厨师，不仅记住每个步骤，还能理解为什么要这么做，甚至能提前想到下一步。它会根据不同的食材和调料，合理安排顺序，确保菜肴味道正宗。通过学习大量菜谱和厨师的思考过程，这个系统能更真实地模拟厨师的决策，帮助厨师改良菜谱或提前预测菜肴的味道。这就像有个聪明的助手，不仅会做菜，还能告诉你为什么这么做，帮你做出更好吃的菜。
ELI14 Explained like you're 14
想象你在学校的食堂里点餐，你喜欢吃披萨，但又不知道今天的披萨会不会好吃。一个普通的机器人可能只会记住你点了披萨，然后告诉你吃了没有。但这个特别的机器人，不仅记住你点了什么，还会想一想：为什么我喜欢这个披萨？它用什么调料？是不是最近的天气影响了我的心情？它会根据你平时喜欢的口味，提前猜到你会不会喜欢今天的披萨。它还会告诉你：我觉得这个披萨味道不错，因为它用了你喜欢的蘑菇和芝士。这个机器人就像一个懂你心思的朋友，不仅知道你喜欢吃什么，还能理解你喜欢的原因，帮你做出更好的选择。这样，你每次点餐都能吃得开心又满意！
Glossary
Hierarchical Context Compression (层次化上下文压缩)
一种将用户历史信息按重要性和类型进行分层压缩的方法，减少噪声，保留决策相关信号。技术上结合了多级筛选和内容摘要。
在论文中用于融合异构多域用户行为历史，确保模型关注关键决策线索。
Trajectory Distillation (轨迹蒸馏)
利用强大预训练模型生成用户思考轨迹，并通过筛选和微调，构建高质量训练数据。技术上结合了软标签和奖励机制。
在训练阶段，用于增强模型对用户深层决策过程的理解。
Structured Prompt Optimization (结构化提示优化)
设计引导模型进行多步骤推理的提示策略，通过自我修正和闭环优化，提高推理准确性。
用于提升模型在折叠上下文中的推理能力，确保输出合理性。
Hybrid Reward (混合奖励)
结合行为匹配奖励和轨迹质量评分的奖励机制，用于强化学习优化。
在模型训练中用于提升模拟的真实性和诊断能力。
Open Questions Unanswered questions from this research
- 1 如何进一步融合多模态信息（如视觉、声音）以丰富用户模型，提升多域场景的泛化能力仍未充分解决。
- 2 奖励机制的主观性和定义的标准化问题，影响模型的稳定性和可迁移性。
- 3 大规模实时应用中模型的效率和成本仍需优化，未来需探索更高效的训练和推理策略。
Applications
Immediate Applications
广告离线评估
利用DASH模拟用户多域行为，帮助广告平台在上线前评估推荐算法效果，减少A/B测试成本，提升模型鲁棒性。
用户行为分析
通过生成用户思考轨迹，分析用户偏好变化，为个性化推荐和内容优化提供深层次的决策线索。
Long-term Vision
多模态多域用户建模
结合视觉、声音等多模态信息，构建更全面的用户画像，实现跨平台、跨场景的个性化推荐。
Abstract
Recent advances in LLM-based user simulation have shown promise for offline evaluation of recommendation and advertising systems. However, existing simulators typically infer user preferences from single-domain interaction histories and are primarily optimized to reproduce observable actions such as clicks. Consequently, they capture only a partial view of user preferences, while action-only prediction easily induces model shortcuts and limits both the fidelity and diagnostic value of simulation. To address these challenges, we propose DASH, a decision-aware user simulator that jointly generates thinking traces and predicts behavioral actions from heterogeneous cross-domain histories. DASH first introduces a Context Engineering stage that folds heterogeneous cross-domain histories into decision-relevant context, together with prompt optimization for effective reasoning over the folded context. To train a user simulator, DASH distills thinking trajectories from strong LLMs as SFT data, and further tailors a rubric-based reward model that evaluates thinking traces along form, content, and logic for RL training. Combined with the action reward, these signals jointly improve action prediction and thinking quality. Extensive experiments on real-world Tencent advertising data spanning five heterogeneous content domains demonstrate the effectiveness, efficiency, fidelity, and diagnostic value of DASH.