SteadyTray: Learning Object Balancing Tasks in Humanoid Tray Transport via Residual Reinforcement Learning
ReST-RL combines pre-trained locomotion policies with residual modules, achieving 96.9% success in stable humanoid tray transport under disturbances.
Key Findings
Methodology
This paper introduces ReST-RL, a hierarchical reinforcement learning framework that decouples locomotion from payload stabilization. The base policy, trained via PPO, ensures stable walking, while a residual module, built with an encoder and adapter, learns corrective actions based on privileged information such as robot and payload states. Two residual architectures—Action Adapter and FiLM—are evaluated. The training incorporates observation delay, control latency, and domain randomization to enhance robustness and transferability. The residual module is trained with privileged data and then distilled into a deployable form, enabling zero-shot sim-to-real transfer. Extensive simulation and real-world tests on Unitree G1 demonstrate high success rates (>96%) and robustness against external pushes and diverse payloads.
Key Results
- In simulation, ReST-RL achieved a success rate of 96.9%, outperforming end-to-end baselines by approximately 8%. It maintained payload uprightness with tilt angles below 0.2 radians during external pushes and disturbances, across various payload sizes and shapes. The approach demonstrated superior robustness, with success rates exceeding 74% in real-world tests involving external forces and different objects such as cups and tools. The inclusion of observation delay further improved stability, indicating enhanced real-world applicability.
- Comparative analysis showed residual modules significantly improved payload stability without sacrificing gait quality. Ablation studies revealed privileged information encoding, observation delay, and domain randomization as critical factors for robustness. The model maintained high performance across diverse payload geometries and external disturbance directions, confirming its generalization capacity. The success in sim-to-real transfer underscores the effectiveness of the training strategies and architecture design.
- The results highlight that structured residual reinforcement learning effectively addresses the conflicting objectives of locomotion and payload stabilization. The approach's modularity and robustness suggest promising applications in service robots, logistics, and healthcare, where dynamic stability during object transport is essential. The ability to generalize across objects and disturbances paves the way for deploying humanoid robots in real-world scenarios with minimal fine-tuning.
Significance
This work advances humanoid robotics by integrating hierarchical reinforcement learning with residual control, enabling robots to carry unsecured, non-rigid payloads during dynamic walking. It addresses longstanding challenges in balancing gait stability with payload uprightness, crucial for real-world applications like delivery, assistance, and medical transport. The modular architecture facilitates high transferability from simulation to real robots, reducing development costs and time. Its robustness against external disturbances and diverse payloads marks a significant step toward autonomous, reliable service robots capable of operating in unstructured environments, thus broadening their practical deployment potential.
Technical Contribution
The paper presents a novel hierarchical RL architecture combining a pre-trained locomotion policy with a residual correction module, trained with privileged information and distilled for deployment. The residual module employs encoder-adapter structures, including FiLM modulation, to learn corrective actions that stabilize payloads without degrading gait. The training incorporates observation delays and domain randomization, enhancing robustness and sim-to-real transfer. This approach differs from traditional end-to-end RL by explicitly decoupling gait and stabilization objectives, providing a flexible, modular control framework with theoretical guarantees of stability and robustness.
Novelty
This is the first application of residual reinforcement learning for humanoid payload stabilization during dynamic walking, especially with non-rigid, unsecured objects. The architecture's decoupling of locomotion and stabilization objectives, combined with privileged information training and distillation, offers a new paradigm for complex bimanual control. Unlike prior work focusing solely on end-to-end policies or upper-body stabilization, this method ensures gait stability while actively canceling gait-induced perturbations, representing a significant innovation in robot control design.
Limitations
- The approach relies heavily on simulated training with extensive domain randomization; real-world environments with unmodeled dynamics or sensor noise may still pose challenges. Extreme external forces or highly deformable payloads could cause failure modes not yet addressed. Computational costs for training residual modules and distillation processes are non-trivial, potentially limiting scalability. The current framework is optimized for specific payload types; generalization to highly complex or multi-object scenarios requires further research.
Future Work
Future directions include integrating adaptive perception systems for better disturbance detection, extending the framework to multi-object and multi-robot scenarios, and optimizing computational efficiency. Exploring learning-based perception modules and multi-modal sensory data could further improve robustness. Additionally, deploying the system in more unstructured environments and diverse real-world applications, such as disaster response or autonomous logistics, will be critical for practical adoption.
AI Executive Summary
Humanoid robots have made significant progress in dynamic locomotion, yet stable payload transport during walking remains a formidable challenge. Traditional control methods often struggle to balance gait stability with payload uprightness, especially under external disturbances. This paper introduces ReST-RL, a hierarchical reinforcement learning framework that explicitly separates locomotion from payload stabilization. The core idea is to leverage a pre-trained gait policy and augment it with a residual correction module trained using privileged information, which actively cancels gait-induced perturbations.
The architecture employs two residual designs—Action Adapter and FiLM—both of which are trained in simulation with domain randomization, observation delay, and disturbance augmentation. The residual modules learn corrective actions that improve payload stability without degrading gait quality. The training process involves a teacher-student paradigm, where privileged information guides the residual module, which is then distilled into a deployable form that relies only on real observations.
Extensive simulation results demonstrate success rates exceeding 96%, with the system effectively resisting external pushes and handling diverse payloads, including fragile objects like wine glasses and medical tools. The approach also exhibits excellent sim-to-real transfer, validated on a Unitree G1 humanoid robot, where payloads remain level during external disturbances and object manipulations. This modular, robust framework significantly advances the deployment of humanoid robots in real-world tasks such as delivery, elder-care, and medical assistance, marking a major step toward autonomous, reliable service robots.
Despite its strengths, the method faces limitations in extreme disturbance scenarios and highly deformable payloads. Future work aims to incorporate adaptive perception, multi-object control, and real-world testing in more unstructured environments, broadening its applicability and robustness for industry deployment.
Deep Dive
Abstract
Stabilizing unsecured payloads against the inherent oscillations of dynamic bipedal locomotion remains a critical engineering bottleneck for humanoids in unstructured environments. To solve this, we introduce ReST-RL, a hierarchical reinforcement learning architecture that explicitly decouples locomotion from payload stabilization, evaluated via the SteadyTray benchmark. Rather than relying on monolithic end-to-end learning, our framework integrates a robust base locomotion policy with a dynamic residual module engineered to actively cancel gait-induced perturbations at the end-effector. This architectural separation ensures steady tray transport without degrading the underlying bipedal stability. In simulation, the residual design significantly outperforms end-to-end baselines in gait smoothness and orientation accuracy, achieving a 96.9% success rate in variable velocity tracking and 74.5% robustness against external force disturbances. Successfully deployed on the Unitree G1 humanoid hardware, this modular approach demonstrates highly reliable zero-shot sim-to-real generalization across various objects and external force disturbances.