Risk-Sensitive Reinforcement Learning Applied to Control under Constraints

Reinforcement learning has evolved from early model-based approaches to model-free algorithms like Q-learning, enabling agents to learn optimal policies through experience. Risk-sensitive RL emerged to address safety concerns, focusing on variance or worst-case scenarios. However, many methods rely on explicit models or assume simplified environments, limiting real-world applicability. Recent advances include probabilistic constraints and utility-based risk measures, but these often depend on strong assumptions about system dynamics or distributional knowledge. The need for model-free, flexible algorithms that can operate in continuous, nonlinear, and uncertain environments remains pressing, especially in industrial control and autonomous navigation.

The core challenge is designing RL algorithms that can guarantee system safety by controlling the probability of entering error states, without relying on explicit models. Existing methods either penalize unsafe states with high negative rewards or impose hard constraints, but these approaches lack flexibility and often fail in complex environments. Moreover, many algorithms cannot handle continuous state spaces or adapt to changing dynamics. The difficulty lies in estimating and controlling the risk of error states directly from experience, while still optimizing system performance. Achieving a balance between safety and reward in a model-free, scalable manner is a key open problem.

The paper introduces a weighted, model-free RL framework that explicitly incorporates the probability of error states as a risk measure. Key innovations include: 1) defining risk as the probability of reaching error states, 2) combining this risk with the reward via a weighted sum, 3) adaptively tuning the weight parameter ξ to satisfy risk constraints, 4) extending Q-learning to estimate both value and risk functions simultaneously, and 5) applying the approach to continuous, nonlinear environments without explicit system models. These innovations enable flexible, safe decision-making in complex systems, overcoming limitations of prior variance-based or model-dependent methods.

• �� Define the set of error states and model the risk as the probability of reaching these states under a policy.
• �� Construct a combined value function Vξ = ξV - ρ, where V is the reward-based value and ρ the risk.
• �� Use Q-learning to estimate Q-values for reward and risk functions, updating estimates based on experience.
• �� Initialize ξ=0, perform learning to minimize risk, then gradually increase ξ to prioritize reward while keeping risk below threshold ω.
• �� Adjust ξ iteratively: after each phase, evaluate risk; if risk exceeds ω, decrease ξ; if within limits, increase ξ.
• �� The algorithm does not require explicit system models, relying solely on observed transitions and rewards.
• �� Convergence conditions are discussed, emphasizing the importance of proper policy classes and discount factors.

Experiments involved stochastic control tasks: a feed tank with uncertain inflows and a grid world environment. Metrics included risk probability, reward, and convergence behavior. Baselines comprised classical Q-learning and chance-constrained methods. Hyperparameters such as learning rate, discount factor, and weight adjustment step size were tuned. Multiple runs tested robustness, with ablation studies analyzing the impact of ξ adjustment. Results demonstrated the algorithm's ability to keep risk below the threshold while improving reward, even under model uncertainties and in continuous spaces.

The algorithm achieved risk probabilities below 0.05 in the feed tank task, with outflow deviations within ±2%, outperforming traditional methods by 15%. In grid world, risk was reduced to 0.02, reward increased by 20%. The approach maintained stable convergence in continuous, nonlinear systems without explicit models, outperforming model-based control in uncertain environments. These results confirm the method's effectiveness and robustness across different scenarios.

Applicable in industrial process control, autonomous vehicles, robotic navigation, and any safety-critical system where risk of failure must be managed. The model-free nature allows deployment in environments with unknown or complex dynamics. It can be integrated with deep neural networks for high-dimensional problems, enabling scalable, safe decision-making in real-world applications.

The convergence speed may slow in high-dimensional or highly nonlinear environments, requiring further optimization. Parameter tuning for ξ adjustment remains challenging. Risk estimation accuracy may degrade under extreme stochasticity, and the theoretical guarantees depend on specific assumptions about policy classes and discount factors. Future work will focus on deep learning integration, faster convergence, and broader applicability.