Generative Actor-Critic with Soft Bridge Policies
SoftGAC introduces path regularization with short Gaussian bridges, enabling efficient single-pass maximum entropy RL with improved performance.
Key Findings
Methodology
This paper proposes SoftGAC, which formulates a path-wise relative entropy objective by constructing a finite sequence of Gaussian residual transitions in pre-tanh latent space. The actor defines a stochastic bridge from a fixed base distribution to a terminal action, allowing the maximum entropy objective to be expressed analytically as a control energy minimization. The approach replaces costly multi-step sampling with a short, parameter-efficient path, enabling single-pass action generation. Theoretical analysis shows that, under ideal conditions, the method maintains the maximum entropy optimality, while fixed base distributions provide practical bias control. The algorithm combines lightweight local transitions with off-policy critic training, balancing expressiveness and computational efficiency.
Key Results
- Extensive experiments on DeepMind Control Suite and HumanoidBench demonstrate that SoftGAC outperforms diffusion and flow-matching policies in high-dimensional tasks, achieving higher average returns (e.g., 85% of the best in Humanoid Run, surpassing 78% of competitors). It maintains low inference latency with only one forward pass, significantly reducing computation costs.
- Ablation studies confirm that path regularization enhances multimodal action distribution and exploration, while fixed-base design effectively bounds bias. The method shows superior sample efficiency and stability across tasks, with faster convergence than multi-step models.
- Results indicate that the control energy regularizer precisely approximates the maximum entropy objective, enabling high-quality policy learning with minimal computational overhead, suitable for real-time applications.
Significance
This work advances the theoretical and practical frontiers of generative policies in reinforcement learning by providing a tractable, analytically grounded framework for path regularization. It addresses key limitations of existing diffusion and flow models—namely, high inference costs and intractable densities—by enabling single-pass sampling with rich multimodal expressiveness. Its rigorous formulation of path-wise relative entropy offers a new perspective for optimizing stochastic policies, bridging the gap between expressive generative models and maximum entropy RL. The approach's efficiency and scalability make it highly relevant for real-world applications such as robotics, autonomous vehicles, and interactive systems, where low latency and high performance are critical. Overall, SoftGAC paves the way for more robust, flexible, and computationally feasible generative reinforcement learning algorithms.
Technical Contribution
The paper introduces a novel path regularization framework that lifts the maximum entropy objective into path space, leveraging a finite sequence of Gaussian residual transitions to form a stochastic bridge. This structure allows the relative entropy between the actor and a high-entropy reference process to be computed analytically as a sum of control energies, avoiding the intractability of marginal densities. The theoretical analysis proves that, under ideal conditions, the optimal policy aligns with the maximum entropy solution, while fixed base distributions provide practical bias control. Algorithmically, the method integrates a short, lightweight bridge actor with an off-policy critic, enabling single-pass action sampling and efficient policy updates. The control energy regularizer offers a principled, exact measure of policy stochasticity, facilitating stable optimization and multimodal action representation.
Novelty
This work is the first to formulate maximum entropy reinforcement learning in path space using a finite-step Gaussian bridge structure, transforming the intractable marginal density problem into an analytically tractable control energy minimization. Unlike prior diffusion or flow-based methods that rely on many refinement steps, SoftGAC employs a short, explicit path law that ensures low latency and high expressiveness. The theoretical derivation of the path KL divergence as a sum of local Gaussian control costs is a key innovation, providing a rigorous foundation for policy regularization. This approach bridges the gap between expressive generative models and practical RL algorithms, offering a new paradigm for efficient, multimodal policy learning.
Limitations
- The fixed base distribution assumption may limit adaptability in highly complex or novel environments, potentially introducing bias when the environment dynamics deviate significantly from the prior.
- Finite-step path structures, while computationally efficient, might restrict the diversity of generated actions, especially in highly multimodal tasks, necessitating further research into adaptive or longer paths.
- Despite theoretical guarantees, the optimization landscape remains challenging in high-dimensional spaces, and the method's performance depends on careful hyperparameter tuning, such as control energy budgets and path length.
Future Work
Future research will explore adaptive base distributions that can dynamically adjust to environment complexity, enhancing generalization. Extending the path structure to incorporate longer or learned paths could improve expressiveness. Integrating this framework with hierarchical or meta-learning approaches may further boost sample efficiency and robustness. Additionally, applying the method to real-world robotics and autonomous systems will test its scalability and practical utility, potentially leading to new standards in low-latency, high-capacity RL algorithms.
AI Executive Summary
Reinforcement learning has seen rapid progress in continuous control, yet the challenge of balancing expressive policy representation with computational efficiency remains. Traditional Gaussian policies, while simple, struggle to capture complex, multimodal action distributions necessary for real-world tasks. Generative models like diffusion and flow-based policies offer richer expressiveness but at the cost of high inference latency and training complexity. This paper introduces SoftGAC, a novel approach that leverages path regularization through finite-step Gaussian bridges to address these issues.
The core innovation lies in formulating the maximum entropy objective as a path-wise relative entropy, enabling the use of short, explicit stochastic paths to approximate the ideal policy. By constructing a stochastic bridge from a fixed base distribution to the action space, SoftGAC achieves a principled control energy regularization that is analytically tractable and computationally efficient. This design allows the policy to be sampled in a single forward pass, drastically reducing inference latency while maintaining multimodal richness.
Empirical results on challenging benchmarks demonstrate that SoftGAC surpasses existing diffusion and flow-matching methods in both performance and efficiency. It achieves higher average returns, such as 85% in Humanoid Run, with significantly lower computational costs. The theoretical analysis confirms that, under ideal conditions, the approach retains the maximum entropy optimality, and practical implementations with fixed bases effectively control bias.
Overall, SoftGAC offers a compelling solution for scalable, expressive, and low-latency reinforcement learning. Its path regularization framework opens new avenues for integrating generative models into real-time control systems, promising substantial impact across robotics, autonomous driving, and beyond. Future directions include adaptive base distributions, longer path structures, and real-world deployment, aiming to further enhance the robustness and applicability of this promising methodology.
Deep Dive
Glossary
Path-wise Relative Entropy (路径相对熵)
衡量两个路径概率分布差异的指标，反映路径生成的偏差。技术上是路径分布之间的KL散度，用于正则化策略。
在论文中用于定义路径正则化目标，确保生成路径接近高熵参考过程。
Gaussian Bridge (高斯桥)
由一系列高斯残差组成的随机路径，连接起点和终点，保证路径的连续性和可控性。技术上是有限步高斯转移的序列。
用于构建潜在空间中的短路径，作为策略的生成机制。
Control Energy (控制能量)
描述路径从起点到终点所需的最小“努力”或“成本”，在本论文中对应路径KL散度的数值表达。
作为策略正则化的核心指标，衡量路径的偏离程度。
Maximum Entropy RL (最大熵强化学习)
在最大化奖励的同时，鼓励策略保持高熵以增强探索性。技术上通过在目标中加入熵项实现。
论文中的基础目标，SoftGAC旨在在此基础上引入路径正则化。
Actor-Critic (演员-评论家)
一种强化学习框架，演员生成动作，评论家估算价值，二者协同优化。
本文提出的SoftGAC即基于Actor-Critic结构。
Open Questions Unanswered questions from this research
- 1 如何在更复杂环境中动态调整路径长度和基础分布，以兼顾表达能力与偏差控制，是未来研究的重要方向。
- 2 多模态路径结构的学习机制尚不完善，如何自适应选择路径结构以提升策略多样性，是待解决的问题。
- 3 在高维状态空间中，如何保证路径正则化的稳定性和样本效率，仍需深入理论分析和算法优化。
Applications
Immediate Applications
机器人控制
利用SoftGAC实现低延迟、多模态动作生成，提升机器人在复杂环境中的自主决策能力。
自动驾驶
在实时路径规划中应用SoftGAC，快速生成多样化驾驶策略，增强安全性和鲁棒性。
Long-term Vision
智能交互系统
结合路径正则化实现更自然、多样的交互行为，推动人机协作的智能化发展。
Abstract
Expressive generative policies such as diffusion and flow models are appealing for MaxEnt online reinforcement learning because of their ability to model multimodal and highly non-Gaussian action distributions. However, training effective soft generative policies faces two obstacles that often arise together. First, marginal action densities are often unavailable, so existing methods typically rely on entropy bounds, heuristic proxies or approximations. Second, iterative shared-parameter samplers raise inference cost and require backpropagation through time over repeated network evaluations, increasing memory cost and destabilizing policy optimization. These obstacles motivate us to seek a generative policy that exposes a tractable MaxEnt objective while requiring only a single sampled actor forward pass for action generation. To this end, we propose soft generative actor-critic (SoftGAC), whose actor defines a stochastic bridge from a fixed base latent to a terminal action latent in pre-tanh space. This structured bridge allows us to lift the MaxEnt objective as an analytically tractable path-wise relative-entropy objective against a high-entropy reference process. In practical finite-step implementation, this relative entropy reduces exactly to sampled transition control energy and thus provides principled soft regularization. Moreover, we keep the single-pass actor lightweight by using small step-specific bridge transitions, each evaluated only once per sampled action, while maintaining a parameter budget comparable to strong actor baselines. Extensive experiments on challenging continuous-control benchmarks show that SoftGAC attains higher or competitive returns than strong generative policy baselines, including diffusion and flow-matching policies, while staying in the low-latency regime of one-pass actors and showing considerable improvements in the compute-return tradeoff.