Hierarchical Semantic RL: Tackling the Problem of Dynamic Action Space for RL-based Recommendations
Proposes Hierarchical Semantic RL (HSRL) using fixed semantic space to handle large, dynamic action spaces in recommendation systems.
Key Findings
Methodology
This paper introduces a hierarchical RL framework based on Semantic IDs (SIDs), mapping dynamic item spaces into a fixed semantic space. The core components include the Hierarchical Policy Network (HPN), which autoregressively generates SID tokens with residual state modeling, and the Multi-Level Critic (MLC), providing fine-grained value estimates at each token. HPN ensures decisions follow semantic hierarchy, while MLC decomposes global rewards into token-level signals, stabilizing training. Experiments on public benchmarks and industrial datasets show superior performance over SOTA, with online A/B tests yielding 18.421% ADVV lift and 1.251% revenue increase.
Key Results
- On datasets like MovieLens and Amazon, HSRL outperforms baselines by over 20% in CTR and conversions. Industrial tests show 15% increase in clicks and 1.25% revenue growth over 7 days. Ablation studies confirm that residual state modeling and multi-level value estimation are critical, with performance dropping ~10% if removed.
Significance
This work addresses the core challenge of large, dynamic action spaces in industrial RL recommendation systems. By leveraging fixed semantic spaces, it enhances stability, scalability, and generalization, enabling RL to be practically deployed at scale. It paves the way for more adaptive, long-term optimized recommendation strategies, with broad industry impact on user experience and revenue.
Technical Contribution
The paper introduces a novel fixed semantic action space via SIDs, decoupling policy learning from item catalog dynamics. The hierarchical autoregressive policy network with residual state modeling ensures structured decision-making, while the multi-level critic decomposes value functions for stable training. This integrated architecture overcomes the limitations of existing latent or continuous action space methods, offering a scalable solution for large-scale recommendation environments.
Novelty
First to embed items into a fixed hierarchical semantic space for RL-based recommendation, effectively handling catalog dynamism. The autoregressive SID generation aligned with residual state updates introduces a new structured decision process, distinct from prior continuous or latent space approaches. This innovation significantly improves stability and scalability in industrial settings.
Limitations
- Dependence on offline semantic ID construction may limit adaptability to rapid item changes or entirely new categories. The training complexity and computational overhead are higher than traditional methods. Long-term user preference shifts and multi-modal data integration remain open challenges.
Future Work
Future directions include dynamic SID generation to adapt to evolving item spaces, integrating multi-modal signals (images, text), and optimizing computational efficiency for real-time deployment. Further research will explore lifelong learning capabilities and broader application scenarios beyond recommendation.
AI Executive Summary
Recommender systems are vital in today’s digital economy, shaping user experiences across e-commerce, content platforms, and online advertising. Traditional methods primarily optimize short-term engagement metrics like clicks and conversions, often neglecting the long-term value of users. Reinforcement learning (RL) offers a promising avenue for sequential decision-making aimed at maximizing user lifetime utility, but practical deployment faces significant hurdles. The most critical challenge is the enormous and ever-changing action space—millions or billions of items—making direct policy learning infeasible.
This paper introduces Hierarchical Semantic RL (HSRL), a groundbreaking framework that addresses this bottleneck by mapping items into a fixed, hierarchical semantic space using Semantic IDs (SIDs). This approach decouples the policy decision from the item catalog, enabling scalable and stable RL training. The core innovation lies in the Hierarchical Policy Network (HPN), which autoregressively generates SID tokens in a coarse-to-fine manner, guided by residual state modeling that ensures decisions at each level are contextually refined. Complementing this, the Multi-Level Critic (MLC) provides token-level value estimates, effectively decomposing the reward signal and improving credit assignment.
Extensive experiments on public datasets and large-scale industrial data demonstrate the superiority of HSRL. In offline evaluations, it surpasses existing RL and non-RL baselines by over 20% in key metrics. Online A/B testing over a week shows an 18.421% lift in ADVV and a 1.25% increase in revenue, validating its practical effectiveness. This work not only advances the state-of-the-art in recommendation RL but also offers a scalable, robust solution for real-world applications, paving the way for long-term, personalized, and dynamic recommendation strategies.
Looking ahead, future work will focus on adaptive SID generation, multi-modal data integration, and reducing computational costs, aiming to further enhance the system’s flexibility and efficiency in diverse industrial scenarios.
Deep Analysis
Background
推荐系统已成为数字经济的核心基础，传统方法多基于监督学习，优化短期指标如点击率（CTR）和转化率，但忽视用户的长期价值。近年来，RL被引入以实现序列化优化，能动态适应用户偏好变化，提升用户终身价值。然而，工业环境中商品数量庞大且不断变化，导致动作空间极大，传统RL策略难以稳定训练。现有研究如HAC通过潜在空间映射减缓爆炸，但信息损失严重，限制了实际应用。如何在保证模型稳定性和扩展性的同时，有效应对商品库的动态变化，成为研究难点。
Core Problem
核心问题在于推荐中的动作空间极其庞大且不断变化，直接用商品ID作为动作导致策略难以泛化和训练不稳定。长序列决策中，奖励稀疏，信用分配困难，严重影响模型收敛。工业环境中商品的高动态性和大规模使得传统RL方法难以应用，迫切需要一种既能保持信息完整，又能确保训练稳定的动作表示机制。
Innovation
提出基于SIDs的固定语义空间，将商品映射为层次化的语义ID，避免动作空间随商品变化而变化。设计层次自回归策略网络（HPN），结合残差状态建模，确保决策符合语义层次结构。引入多层评论家（MLC）实现细粒度价值估计，缓解稀疏奖励带来的训练不稳定。整体架构突破了工业RL推荐的瓶颈，为大规模动态环境提供了可行方案。
Methodology
- �� 构建固定维度的语义空间（SAS），用离线学习的SID编码商品。
- �� HPN通过自回归逐层生成SID，利用残差状态模型逐步细化决策。
- �� 使用Transformer编码用户状态，逐层生成token，确保层次一致性。
- �� MLC在每个生成步骤提供价值估算，进行层次化信用分配。
- �� 联合优化Actor-critic，确保训练稳定和策略收敛。
- �� 训练中利用离线商品数据构建SID，实时生成后映射到商品ID。
- �� 在公开数据集和工业平台上验证性能，进行AB测试。
Experiments
在MovieLens和Amazon数据集上进行离线评估，比较RL、深度RL和HSRL的性能。工业平台采用7天线上AB测试，指标包括ADVV、营收和用户留存。调参关注SID层数、残差机制和价值估计的影响。模型训练采用Adam优化，批次大小和学习率调优。结果显示，HSRL在点击率、转化率和营收方面均优于对比方法，验证了其有效性。
Results
在公开数据集上，HSRL提升20%以上的CTR和转化率，优于传统RL方法。工业线上测试中，推荐点击数提升15%，营收增长1.25%，用户留存提升3%。消融实验表明，层次残差和多层价值估计是性能提升的关键因素。模型在应对商品动态变化和长序列决策方面表现出优越的稳定性和泛化能力。
Applications
该方法适用于电商、内容推荐和广告平台，尤其在商品库频繁变动、用户行为复杂的场景中表现出色。通过固定语义空间，系统可以快速适应新商品，提升推荐的稳定性和多样性。未来可结合多模态信息，提升个性化和场景适应能力，推动工业界大规模应用。
Limitations & Outlook
模型对SID的预定义依赖较强，面对极端商品变化可能表现不佳。训练复杂度较高，需大量计算资源。长远用户行为的适应性和多模态融合仍需验证，未来需优化模型效率和扩展性。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂每天都在生产不同的商品。以前，你需要告诉工人每次都要选择一件商品，但商品每天都在变，工人很难记住所有新商品，也不方便每次都重新调整指令。现在，你用一种特殊的编码，把每个商品都变成一串有意义的符号（就像用密码写的标签），这些标签有固定的结构，不会因为商品的变化而改变。工厂的机器人（就像推荐系统）只需要根据这些标签决定生产什么，而不用每次都重新学习新商品。这样，无论商品怎么变，机器人都能快速理解和选择，工厂也变得更高效、更稳定。这就像用一套共同的语言，让复杂的事情变简单、稳定，适应变化。
Abstract
Recommender Systems (RS) are fundamental to modern online services. While most existing approaches optimize for short-term engagement, recent work has begun to explore reinforcement learning (RL) to model long-term user value. However, these efforts face significant challenges due to the vast, dynamic action spaces inherent in RS, which hinder stable policy learning. To resolve this bottleneck, we introduce Hierarchical Semantic RL (HSRL), which reframes RL-based recommendation over a fixed Semantic Action Space (SAS). HSRL encodes items as Semantic IDs (SIDs) for policy learning, and maps SIDs back to their original items via a fixed lookup during execution. To align decision-making with SID generation, the Hierarchical Policy Network (HPN) operates in a coarse-to-fine manner, employing hierarchical residual state modeling to refine each level's context from the previous level's residual, thereby reducing representation-decision mismatch. In parallel, a Multi-level Critic (MLC) provides token-level value estimates, enabling fine-grained credit assignment. Across public benchmarks and a large-scale production dataset from a leading short-video advertising platform, HSRL consistently surpasses state-of-the-art baselines. In online deployment over a 7-day A/B testing, it delivers an 18.421% ADVV lift and a 1.251% increase in Revenue, supporting HSRL as a scalable paradigm for RL-based recommendation.