GCImOpt: Learning efficient goal-conditioned policies by imitating optimal trajectories

In the field of control, designing optimal control policies to achieve task objectives while minimizing costs has been a long-standing challenge. Traditional imitation learning methods rely on expert demonstration data, which is often costly to collect and may not be ideal. Additionally, designing optimal closed-loop controllers for many dynamical systems is very difficult or even impossible. To address these issues, trajectory optimization is widely used in practice, such as in model predictive control (MPC). However, while MPC allows the design of near-optimal closed-loop controllers, solving optimization problems at high frequency makes it computationally expensive. In recent years, researchers have begun exploring learning efficient control policies by imitating optimal trajectories, which has been widely applied in aerospace and other fields.

For many dynamical systems, finding an optimal closed-loop controller or policy is very difficult. Traditional imitation learning methods rely on expert demonstration data, which is often costly to collect and may not be ideal. Additionally, existing GCRL methods require complex reward shaping and online environment interaction, increasing the difficulty of exploration. To address these issues, this paper proposes a new method, GCImOpt, which learns efficient goal-conditioned policies by imitating optimal trajectories.

The core innovations of the GCImOpt method lie in its simple yet efficient data generation and policy training process. • Trajectory Optimization: Generates high-quality demonstration datasets through trajectory optimization, avoiding the expensive and potentially suboptimal demonstration data collection in traditional imitation learning. • Data Augmentation: Uses data augmentation to treat intermediate states as goals, significantly increasing the training dataset size. • Policy Training: Trains goal-conditioned neural network policies on the generated datasets, demonstrating the method's generality. • Computational Efficiency: Uses the FATROP solver for fast parallel dataset generation, significantly enhancing data generation efficiency.

The implementation of the GCImOpt method includes the following steps: • Dataset Generation: Generates high-quality demonstration datasets through trajectory optimization. Uses the FATROP solver for fast parallel dataset generation. • Data Augmentation: Uses data augmentation to treat intermediate states as goals, significantly increasing the training dataset size. • Policy Training: Trains goal-conditioned neural network policies on the generated datasets, using a multi-layer perceptron (MLP) structure. • Policy Evaluation: Evaluates the success rate and efficiency of the policies in a simulated environment, verifying their performance across different control tasks.

The experimental design includes evaluating the GCImOpt method on four different continuous control tasks: cart-pole system, planar quadrotor, three-dimensional quadrotor, and 6-DoF robot arm. The safe-control-gym and urdf2casadi libraries are used for system modeling and simulation. In the experiments, the success rate and efficiency of the policies are evaluated through closed-loop control tasks in a simulated environment. The results show that GCImOpt policies demonstrate high success rates and near-optimal control profiles across multiple tasks.

The experimental results show that GCImOpt policies demonstrate high success rates and near-optimal control profiles across multiple tasks. In the cart-pole system, GCImOpt policies achieved a success rate of 94.83% with an average relative error of 27.252%. For the planar quadrotor task, a 128-unit MLP policy achieved a success rate of 99.77% with a relative cost error of 5.282%. In the 3D quadrotor task, a 128-unit MLP policy achieved a success rate of 97.8% with a relative cost error of 60.145%. These results indicate that GCImOpt policies not only perform excellently across multiple control tasks but can also be deployed on resource-constrained controllers, significantly reducing computational costs.

The GCImOpt method performs excellently across multiple control tasks and has broad application prospects. • Cart-Pole System: Used for balance control tasks. • Quadrotors: Used for stabilization and navigation tasks. • Robot Arm: Used for precise point-reaching tasks. GCImOpt policies demonstrate high success rates and near-optimal control profiles in these tasks, particularly suitable for applications requiring rapid response and low computational overhead.

While GCImOpt performs excellently across multiple tasks, more task-specific tuning and richer datasets may be needed for certain complex tasks. Additionally, GCImOpt shows high relative cost errors in the 3D quadrotor task, indicating limited policy efficiency. This may be due to the sensitivity of quadrotor dynamics, requiring further optimization of dataset coverage. Future research directions include further optimizing dataset generation and policy training processes to enhance policy efficiency and success rates.