Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection
Learn feasible inventory policies via differentiable projection, achieving an average optimality gap below 1%.
Key Findings
Methodology
This study proposes embedding a differentiable convex optimization module within the policy. Specifically, a neural network generates continuous action targets, a quadratic program projects them onto the relaxed feasible set, and a dual-informed integer mapping restores integrality while preserving feasibility. The policy trains end-to-end on a differentiable simulator using pathwise gradients to handle hard constraints.
Key Results
- On small instances, the method achieves an average optimality gap below 1%. In larger networks, it outperforms state-of-the-art echelon base-stock policies by up to 9.75% and a rolling-horizon multi-stage stochastic program by at least 7.7%.
- In an industry-scale case study from ASML, the method reduces average cost by 3.22%, particularly excelling in tightly capacitated systems with high demand variability.
- Ablation studies show that ignoring projection sensitivities causes training divergence, and replacing the dual-informed integer map with naive integer mapping increases cost.
Significance
This study demonstrates the economic benefits of deep reinforcement learning in sequential decision problems with interdependent hard constraints, particularly excelling where planning is hardest and most relevant. It provides a novel solution to complex constraint problems where traditional inventory policies struggle.
Technical Contribution
Technical contributions include a policy architecture combining a differentiable decision rule, convex projection, and dual-informed integer mapping, offering new theoretical guarantees and ensuring the entire feasible action space is reachable without per-state integer programming.
Novelty
This method is the first to integrate differentiable projection with deep reinforcement learning for feasible policy learning under complex constraints, offering significant innovation in handling interdependent hard constraints compared to existing methods.
Limitations
- The method may incur high computational costs in high-dimensional settings, especially in complex networks.
- It requires a differentiable simulator for end-to-end training, which may not be feasible in some applications.
Future Work
Future research could explore applying this method to larger instances and investigate training without a differentiable simulator. Further optimization of computational efficiency is also an important direction.
AI Executive Summary
Many operational problems are constrained sequential decision processes with large combinatorial action spaces and interdependent feasibility constraints. Mixed-integer linear programs (MILPs) handle such constraints flexibly but scale poorly in stochastic environments. Deep reinforcement learning (DRL) promises scalable decision rules, but existing methods either penalize constraints rather than enforce them or rely on feasibility mechanisms that break down once constraints interact.
This paper bridges the gap by embedding a differentiable convex optimization module inside the policy: a neural network proposes continuous action targets, a quadratic program projects them onto the relaxed feasible set, and a dual-informed integer mapping restores integrality while preserving feasibility. Given a differentiable simulator, the policy trains end-to-end from sampled trajectories using pathwise gradients, while handling hard constraints with similar flexibility to MILPs.
We apply the method to multi-echelon production-inventory planning under shared resource and material constraints. Our policy attains an average optimality gap below 1% on small instances. It further outperforms state-of-the-art echelon base-stock policies by up to 9.75% and a rolling-horizon multi-stage stochastic program by at least 7.7% in larger networks. On an industry-scale case study from ASML, it reduces average cost by up to 3.22%. The savings are largest where planning is hardest: in tightly capacitated systems with high demand variability. More broadly, our work shows that DRL can deliver economically significant savings in sequential decision problems with interdependent hard constraints, which are widespread in practice.
Deep Analysis
Background
The multi-echelon inventory planning problem is a classic research area in operations research, involving coordination of inventory and production decisions across multiple supply chain levels. Traditional methods like mixed-integer linear programming (MILP) perform well in handling complex constraints but scale poorly in uncertain and large-scale problems. Recently, deep reinforcement learning (DRL) has gained attention for its decision-making capabilities in uncertain environments, yet it still faces challenges in handling complex constraints.
Core Problem
The core problem addressed in this paper is learning feasible inventory policies in sequential decision problems with interdependent hard constraints. Traditional methods are inefficient in handling these complex constraints, while existing DRL methods either fail to strictly enforce constraints or break down when constraints interact.
Innovation
The core innovation of this paper is embedding a differentiable convex optimization module within DRL policies to learn feasible strategies under complex constraints. Specifically, a neural network generates continuous action targets, a quadratic program projects them onto the relaxed feasible set, and a dual-informed integer mapping restores integrality. This approach offers significant advantages in handling interdependent hard constraints.
Methodology
- �� A neural network generates continuous action targets.
- �� A quadratic program projects targets onto the relaxed feasible set.
- �� A dual-informed integer mapping restores integrality.
- �� End-to-end training using pathwise gradients on a differentiable simulator.
Experiments
The experimental design includes validating the method in multi-echelon production-inventory planning problems. Benchmarks include state-of-the-art echelon base-stock policies and rolling-horizon multi-stage stochastic programs. The experiments measure the performance of different methods on small and large instances, with a focus on performance in tightly capacitated systems with high demand variability.
Results
Experimental results show that the method achieves an average optimality gap below 1% on small instances. In larger networks, it outperforms state-of-the-art echelon base-stock policies by up to 9.75% and a rolling-horizon multi-stage stochastic program by at least 7.7%. In an industry-scale case study from ASML, the method reduces average cost by 3.22%.
Applications
This method can be directly applied to multi-echelon production-inventory planning, particularly in tightly capacitated systems with high demand variability. Its impact on the industry lies in achieving more efficient inventory management under complex constraints.
Limitations & Outlook
While the method performs well in handling complex constraints, it may incur high computational costs in high-dimensional settings. Additionally, the method relies on a differentiable simulator, which may not be feasible in some applications. Future research could explore training without a differentiable simulator and further optimize computational efficiency.
Plain Language Accessible to non-experts
Imagine you're in a kitchen preparing a meal. You have various ingredients, but each is limited in quantity. You need to prepare several dishes within a limited time, each requiring different ingredients. You must decide how much of each ingredient to use for each dish, ensuring you don't exceed the total available. This is like the problem in the paper: making optimal decisions under limited resources. Traditional methods are like a cookbook, guiding you but possibly lacking flexibility in uncertainty. This paper's method is like a smart assistant, providing the best advice based on the current situation, ensuring every dish is completed smoothly.
ELI14 Explained like you're 14
Imagine you're playing a game where you need to complete multiple tasks within a limited time, each requiring different resources like coins and energy. You have to smartly allocate these resources to ensure each task is completed. Traditional methods are like a guidebook, telling you what to do, but might not be flexible in unexpected situations. This paper's method is like a super-smart game assistant, giving you the best advice based on the game's progress, ensuring you complete all tasks on time and score high!
Glossary
Deep Reinforcement Learning
A method using deep neural networks to optimize sequential decisions under uncertainty.
Used to optimize decisions in multi-echelon inventory planning.
Mixed-Integer Linear Programming
A method for solving optimization problems with integer and continuous variables under linear constraints.
Traditional method for handling complex constraints.
Differentiable Optimization
An optimization method allowing gradient propagation, enabling optimization through automatic differentiation.
Used to embed a differentiable convex optimization module in the policy.
Quadratic Programming
An optimization method where the objective function is quadratic and constraints are linear.
Used to project continuous action targets onto the relaxed feasible set.
Dual-Informed Integer Mapping
A mapping method using the dual variables of the projection to prioritize items.
Used to restore integrality while preserving feasibility.
Open Questions Unanswered questions from this research
- 1 How to train without a differentiable simulator? Current methods rely on a differentiable simulator for end-to-end training, which may not be feasible in some applications.
Applications
Immediate Applications
Multi-Echelon Inventory Management
This method can optimize inventory and production decisions in multi-echelon supply chains, especially in systems with high demand variability and tight capacities.
Long-term Vision
Complex Constraint Optimization
The method can be extended to other complex constraint optimization problems like traffic management and energy distribution, providing more efficient solutions.
Abstract
Many operational problems are constrained sequential decision processes with large, combinatorial action spaces and interdependent feasibility constraints. Mixed-integer linear programs (MILPs) handle such constraints flexibly but scale poorly in stochastic environments. Deep reinforcement learning (DRL) promises scalable decision rules, but existing methods either penalize constraints rather than enforce them, or rely on feasibility mechanisms that break down once constraints interact. We bridge this gap by embedding a differentiable convex optimization module inside the policy: a neural network proposes continuous action targets, a quadratic program projects them onto the relaxed feasible set, and a dual-informed integer mapping restores integrality while preserving feasibility. Given a differentiable simulator, the policy trains end to end from sampled trajectories using pathwise gradients, while handling hard constraints with similar flexibility to MILPs. We show that our feasibility enforcement has bounded error relative to an exact integer projection and ensures the entire feasible action space is reachable. We apply the method to multi-echelon production-inventory planning under shared resource and material constraints. Our policy attains an average optimality gap below 1% on small instances. It further outperforms state-of-the-art echelon base-stock policies by up to 9.75% and a rolling-horizon multi-stage stochastic program by at least 7.7% in larger networks. On an industry-scale case study from ASML, it reduces average cost by up to 3.22% relative to the best-known benchmark policy. The savings are largest where planning is hardest: in tightly capacitated systems with high demand variability. More broadly, our work shows that DRL can deliver economically significant savings in sequential decision problems with interdependent hard constraints, which are widespread in practice.