Soft Forward-Backward Representations for Zero-shot Reinforcement Learning with General Utilities

Reinforcement learning (RL) has seen rapid development with deep neural networks, exemplified by algorithms like DQN, A3C, and SAC, which excel in environments with well-defined rewards. However, these methods depend heavily on explicit reward signals and struggle with tasks involving sparse, complex, or unknown reward functions. Zero-shot RL aims to leverage offline datasets and pre-trained policies to generalize to new tasks without retraining. Early approaches like goal-conditioned RL and representation learning provided some flexibility but were limited to linear reward structures. Forward-backward (FB) algorithms introduced a way to represent policies via occupancy measures, capable of solving linear reward problems efficiently. Nonetheless, their expressiveness was restricted, unable to handle non-linear utilities such as distribution matching or pure exploration. Recent advances in maximum entropy RL introduced stochastic policies with broader support but still faced limitations in general utility optimization. This paper builds upon these foundations, proposing a soft variant of FB that overcomes these limitations, enabling direct optimization of complex utilities.

Existing RL algorithms are constrained by the assumption of linear rewards, which restricts their capacity to handle tasks like distribution matching, pure exploration, or constrained optimization. The FB algorithm, while effective for linear rewards, cannot guarantee solutions for non-linear, differentiable utilities. This gap limits the applicability of zero-shot RL in real-world scenarios requiring flexible objective formulations. The core challenge is to develop a method that can learn a diverse set of policies offline and efficiently identify the optimal policy for arbitrary utilities during testing, without iterative gradient-based optimization. Addressing this problem is crucial for advancing autonomous systems capable of adaptive, goal-agnostic behaviors in complex environments.

The paper's key innovations include: 1) Introducing a maximum entropy (soft) variant of the FB algorithm that learns a family of stochastic policies supporting diverse behaviors; 2) Employing low-rank decomposition of occupancy measures into low-dimensional embeddings, facilitating efficient policy search; 3) Training policies offline using maximum entropy objectives, ensuring broad support; 4) Using low-dimensional zero-order search at test time to directly optimize arbitrary differentiable utilities; 5) Combining sample-based and generative model-based inference for continuous spaces, enhancing policy approximation. These innovations collectively enable the algorithm to handle a wider class of RL problems beyond linear rewards, with theoretical guarantees of near-optimality.

• �� Construct a family of parameterized policies {πz}z∈Rd, with occupancy measures decomposed as M z = F⊤z B; • Incorporate maximum entropy regularization during training to promote stochasticity and full support; • Train maximum entropy policies offline using dataset of transitions, optimizing the policy via Bellman residual minimization; • During inference, perform low-dimensional zero-order search (e.g., random sampling, CEM) over embedding z to maximize the target utility f(Mπ); • Approximate occupancy measures using sample-based importance sampling or generative models, enabling continuous space inference; • Validate the approach across tasks like goal-reaching, exploration, and constrained RL, comparing offline evaluation metrics; • Use hyperparameters such as embedding dimension, entropy coefficient, and search iterations to tune performance.

Experiments involved low-dimensional goal-reaching tasks and high-dimensional deep RL benchmarks like DeepMind Control Suite. In simple environments, the method achieved 100% success, with policies becoming more deterministic as the embedding norm increased. In complex tasks, the approach outperformed baseline linear RL and FB variants, with offline scores exceeding 15% improvements in reward. The experiments tested different utility functions, including distribution matching, goal-reaching, and exploration, demonstrating robustness and scalability. Hyperparameters like embedding size and search iterations were tuned for optimal performance. Results confirmed that Soft FB effectively balances policy diversity, utility maximization, and computational efficiency, validating its practical applicability.

Soft FB retrieved a diverse set of policies supporting complex tasks, outperforming traditional FB and linear RL in offline evaluations. In low-dimensional environments, it achieved perfect goal-reaching success, while in high-dimensional benchmarks, it surpassed baseline scores significantly. The low-dimensional search effectively identified near-optimal policies, reducing computational costs. The method's ability to handle distribution matching and pure exploration tasks was validated through offline metrics, with average reward improvements of over 15%. The experimental results confirmed the theoretical guarantees, demonstrating the method's scalability and robustness across diverse RL scenarios.

This approach is suitable for robotics, autonomous vehicles, recommendation systems, and adaptive control, especially where offline data is abundant but reward functions are complex or unknown. It enables rapid adaptation to new tasks without retraining, reducing deployment time and computational costs. The framework can be extended to multi-agent systems and real-time applications by integrating online fine-tuning and transfer learning, making it highly relevant for industry and research in autonomous decision-making.

The reliance on offline data coverage limits performance when data is sparse or biased. Zero-order search may become computationally expensive in very high-dimensional embedding spaces. Approximation errors from sampling or generative models can affect policy quality, especially in continuous domains. Future work should focus on improving sample efficiency, scalable search algorithms, and online adaptation to address these limitations and enhance real-world deployment.