DoublyAware: Dual Planning and Policy Awareness for Temporal Difference Learning in Humanoid Locomotion

Humanoid robot control has evolved from rule-based systems to learning-based approaches, notably reinforcement learning (RL) and model predictive control (MPC). Early methods relied on predefined trajectories, but recent advances incorporate deep RL and TD-MPC to handle high-dimensional control tasks. However, environmental stochasticity and model inaccuracies pose significant challenges, especially in contact-rich, dynamic environments. Prior works, such as PILCO, PETS, and recent TD-MPC variants, have attempted to address uncertainty, but often treat it as a single source, limiting robustness. The need for explicit uncertainty decomposition and risk-aware planning remains pressing, particularly for real-world deployment where safety and efficiency are critical.

Despite progress, current methods struggle with the dual challenges of environmental randomness and model uncertainty in high-dimensional humanoid control. Random contact interactions and sensory noise induce aleatoric uncertainty, while incomplete environment modeling leads to epistemic uncertainty. Existing approaches either ignore this decomposition or lack robust statistical guarantees, resulting in unstable learning and unsafe behaviors. The core problem is how to effectively quantify, filter, and incorporate these uncertainties into the planning and learning processes, ensuring both safety and efficiency in complex, contact-rich tasks.

The paper introduces three key innovations: 1) embedding conformal prediction into TD-MPC to statistically filter trajectories with risk guarantees, 2) developing group-relative policy constraints (GRPC) to stabilize policy updates via relative advantage estimation, and 3) integrating these into a unified framework that explicitly decomposes and manages planning and policy uncertainties. This approach ensures probabilistic safety, improves sample efficiency, and enhances motion feasibility. Unlike prior works that treat uncertainty monolithically, this method provides a structured, theoretically grounded solution for high-dimensional control.

• �� Trajectory Sampling: Combine policy-guided prior trajectories with exploration via MPPI, sampling in latent space. • Risk Filtering: Use conformal prediction to evaluate trajectory costs, compute nonconformity scores, and filter high-risk trajectories based on risk level α. • Statistical Guarantees: Construct distribution-free prediction sets, ensuring at least (1−α) coverage probability. • Policy Stabilization: Apply group-relative advantage estimation, compute softmax-weighted scores for action groups, and optimize policy within a trust region via KL constraints. • Training Loop: Iterative sampling, filtering, and policy update, leveraging value and model networks for short-term and long-term objectives. • Integration: Combine risk-aware trajectory filtering with stable policy optimization for robust learning.

Experiments conducted on Unitree H1-2 humanoid across diverse locomotion tasks, including standing, walking, sitting, navigating poles, and complex terrains. Baselines include SAC, BC-SAC, AWAC, TD-MPC2. Metrics include cumulative reward, convergence speed, and motion feasibility. Hyperparameters: horizon 3, batch size 256, learning rate 0.0003, prior trajectories 24, group number 3, conformal risk level 0.05. Ablation studies evaluate the impact of conformal filtering and GRPC separately. Results show that DoublyAware achieves faster convergence, higher rewards, and more physically consistent motions, especially in complex tasks like stair climbing and obstacle navigation.

DoublyAware demonstrates approximately 30% improvement in sample efficiency and 20% faster convergence over RL baselines. It maintains higher success rates in complex tasks, with reward increases exceeding 15%. Ablation results confirm that conformal trajectory filtering reduces unsafe exploration, while GRPC enhances policy stability. Motion analysis reveals more kinodynamically feasible trajectories, with fewer falls or unnatural behaviors. These findings validate the effectiveness of structured uncertainty handling in high-dimensional humanoid control.

Immediate applications include autonomous humanoid robots in industrial inspection, service tasks, and disaster response, where safety and data efficiency are paramount. The framework can be integrated into existing control architectures with latent dynamics models, providing risk-aware planning and stable policy learning. Long-term, this approach could enable robots to operate reliably in unstructured, unpredictable environments, reducing human supervision and enhancing autonomy in complex scenarios.