Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning

Robotic systems capable of autonomously performing complex, multi-stage tasks have long been a goal in AI research. Traditional reinforcement learning approaches excel at short-term skills but struggle with long-horizon, multi-step operations due to exploration difficulties and the need for detailed task segmentation. Hierarchical reinforcement learning (HRL) offers a promising framework by introducing temporal abstraction, yet practical challenges such as skill discovery and reward design have limited its success.

This paper introduces Relay Policy Learning (RPL), a novel approach that combines imitation learning from unstructured human demonstrations with reinforcement fine-tuning to address these challenges. The core idea is to leverage a data relabeling algorithm that generates goal-conditioned datasets at both high and low levels of a hierarchical policy. This enables the training of a goal-conditioned architecture where a high-level policy sets subgoals, and a low-level policy executes actions within a fixed horizon, simplifying long-term planning.

The methodology involves two key phases. First, Relay Imitation Learning (RIL) pre-trains the hierarchical policies using unstructured demonstrations, without requiring explicit task segmentation or goal labels. The relabeling process considers all states reachable within a fixed window as potential goals, greatly increasing data diversity and generalization. Second, reinforcement learning via natural policy gradients fine-tunes these policies in simulation, significantly boosting task success rates.

Experimental results in a MuJoCo kitchen environment demonstrate the effectiveness of RPL. The success rate on 17 complex manipulation tasks increased from around 8.8% with pure imitation to over 70% after fine-tuning. The hierarchical structure and data relabeling contributed to superior sample efficiency, robustness, and scalability compared to flat or traditional HRL methods.

Overall, RPL provides a scalable, general framework for long-horizon robotic tasks, reducing reliance on manual task segmentation and dense rewards. Its ability to utilize unstructured demonstration data and improve through environment interaction marks a significant step forward in autonomous robot learning, with broad implications for industrial automation, service robots, and beyond. Despite some computational costs and data dependence, its promising results suggest a bright future for hierarchical, imitation-augmented reinforcement learning in complex real-world scenarios.