PlanRL: A Trajectory Planning Architecture for Reinforcement Learning-based Driving Experts
PlanRL integrates Frenet-based polynomial trajectory planning with RL, achieving 5% score improvement and 19% success rate increase on CARLA benchmarks.
Key Findings
Methodology
This paper introduces a trajectory planning architecture combining RL policy outputs with polynomial trajectory generation within a Frenet coordinate framework. The RL policy, trained via PPO, takes BEV semantic segmentation and vehicle state inputs, producing high-level commands: longitudinal acceleration and lateral offset targets. These targets are processed through a kinematic feasibility check, which adjusts terminal states to ensure physical plausibility. The trajectory generation employs a quadratic polynomial for longitudinal motion and a quintic polynomial for lateral motion, both constrained by vehicle kinematics. This structure reduces environmental complexity, enhances interpretability, and minimizes cumulative tracking errors. The system is trained with dense rewards emphasizing safety, smoothness, and compliance, and validated on CARLA's offline benchmarks, demonstrating superior performance over control-based RL methods.
Key Results
- On CARLA Offline v1, the proposed method improves driving score by 5%, success rate by 8%, and reduces infractions compared to baselines such as Roach and rule-based methods. In NoCrash, success rate increases by 19%, with collision rates significantly lowered. Ablation studies confirm the importance of Frenet coordinate transformation and kinematic checks. The approach maintains high robustness in high-curvature urban scenarios, outperforming existing control-centric RL models, and demonstrates strong generalization across diverse traffic conditions.
Significance
This work advances RL-based autonomous driving by providing a structured, interpretable, and physically feasible path planning framework. It addresses core issues of control command opacity, environmental complexity, and error accumulation, paving the way for safer, more reliable, and more transparent autonomous systems. Its success on CARLA benchmarks suggests strong potential for real-world deployment, especially in urban environments where complex road geometries and traffic interactions are prevalent. The integration of polynomial trajectory planning within RL marks a significant step toward bridging learning-based and classical planning paradigms, offering a scalable solution for industry adoption.
Technical Contribution
The paper proposes a novel hybrid architecture that combines RL policy outputs with polynomial trajectory generation in the Frenet coordinate system, incorporating a kinematic feasibility check to ensure physically plausible paths. It innovatively parameterizes continuous high-level actions using Beta distributions, improving stability and exploration. The approach reduces environmental complexity, enhances interpretability, and minimizes error accumulation, providing a robust, structured framework for RL in autonomous driving. This integration of classical trajectory modeling with modern RL training constitutes a significant technical advancement, enabling safer and more reliable path planning.
Novelty
This study is the first to embed a Frenet-based polynomial trajectory planning module within an RL framework for autonomous driving benchmarks like CARLA. Unlike prior control-output RL models, it introduces a structured, interpretable path representation that directly encodes vehicle kinematics. The combination of high-level RL commands, polynomial path generation, and feasibility checks forms a unique, cohesive architecture that significantly improves robustness and transparency, addressing longstanding challenges in RL-based autonomous navigation.
Limitations
- The model heavily relies on high-quality perception inputs; sensor noise or perception failures could degrade performance. Its robustness under extreme weather or highly dynamic environments remains untested. Computational complexity, especially for real-time deployment, needs further optimization. Additionally, the current validation is primarily in simulation; real-world transfer and robustness require further investigation.
Future Work
Future directions include integrating multi-modal perception to enhance robustness, optimizing trajectory generation for real-time deployment, and extending the framework to real vehicle platforms. Further research will explore end-to-end learning of perception and planning, as well as adaptation to diverse weather and traffic conditions. Bridging the gap between simulation and real-world deployment remains a key goal, aiming for safer, more reliable autonomous driving systems.
AI Executive Summary
Autonomous driving has seen rapid advances, yet many RL-based models still struggle with interpretability and robustness. Traditional RL approaches often directly output low-level control commands, which, while responsive, lack transparency and are prone to error accumulation, especially on complex urban roads. This limits their deployment in real-world scenarios where safety and explainability are paramount.
To address these issues, the authors propose PlanRL, a novel architecture that integrates reinforcement learning with polynomial trajectory planning within the Frenet coordinate system. The core idea is to have the RL policy generate high-level commands—longitudinal acceleration and lateral offset targets—that are then converted into smooth, kinematically feasible trajectories. These trajectories are generated using quadratic and quintic polynomials, ensuring smoothness and adherence to vehicle dynamics. The system also incorporates a kinematic feasibility check, adjusting terminal states to prevent paths from exceeding physical limits, thereby reducing cumulative errors.
Experimental validation on CARLA's offline benchmarks demonstrates the effectiveness of this approach. The proposed method outperforms existing control-based RL models, achieving a 5% increase in driving score and an 8% rise in success rate on the Offline v1 leaderboard. In the NoCrash benchmark, success rates improve by 19%, with fewer infractions and collisions. These results highlight the architecture’s robustness, especially in high-curvature and dense traffic scenarios, confirming its potential for real-world applications.
This work significantly contributes to the field by providing a structured, interpretable, and physically grounded RL framework for autonomous navigation. It bridges the gap between learning-based and classical planning methods, offering a scalable solution that enhances safety, reliability, and transparency. Future work aims to incorporate multi-modal perception, optimize real-time performance, and validate on physical vehicles, moving closer to practical deployment in urban autonomous driving systems.
Deep Analysis
Background
自动驾驶技术经历了从规则驱动到学习驱动的演变。早期依赖手工设计规则，代表如CARLA Autopilot，虽在特定环境表现良好，但缺乏适应性。近年来，RL方法崭露头角，能通过与环境交互自主学习策略，提升复杂场景下的表现。代表工作如Roach、CaRL等，采用轨迹规划或奖励设计优化路径，但多为控制命令输出，缺乏结构化路径表达，难以应对高曲率道路和复杂交通。现有方法在模拟环境中取得一定成功，但在实际应用中仍面临可解释性差、误差累积大等挑战。
Core Problem
传统RL自主驾驶模型多直接输出控制指令，导致路径缺乏可解释性，难以验证决策合理性。同时，缺乏结构化的道路几何表达，模型难以学习复杂道路形状，尤其在高曲率弯道表现不佳。此外，控制输出模型易受累积误差影响，影响路径跟踪精度。这些问题限制了RL在真实复杂交通环境中的应用潜力，亟需引入更具结构化的路径表示和运动学约束机制，以提升系统的稳健性和可解释性。
Innovation
本研究提出将RL策略输出高层指令转化为多项式轨迹，结合Frenet坐标系简化道路几何，显著提升路径表达的结构化和可解释性。引入运动学可行性检测，确保路径在车辆极限内，减少误差累积，增强鲁棒性。采用Beta分布参数化连续动作空间，提升策略稳定性和探索效率。整体架构实现了路径的平滑性、合理性与可控性，为RL自主驾驶提供了新颖的技术路径，突破了传统控制导向模型的局限。
Methodology
- �� 输入：BEV语义分割图像（包括道路、车道线、交通标志等）和车辆状态向量。• RL策略：基于PPO训练，输出纵向加速度与横向偏移目标，采用Beta分布参数化，确保动作范围受控。• 轨迹生成：纵向采用二次多项式描述，横向采用五次多项式，结合边界条件计算系数。• 运动学检测：根据偏移目标，计算最大可达横向位移，调整终点状态，确保路径物理可行。• 轨迹优化：在保证运动学约束下，生成平滑路径，减少误差累积。• 训练：利用密集奖励信号，结合速度、偏差、转向变化和违规惩罚，优化策略。• 评估：在CARLA平台上进行多场景测试，验证路径合理性与鲁棒性。
Experiments
采用CARLA 0.9.10.1版本，测试在离线排行榜v1和NoCrash场景中。指标包括驾驶得分、成功率、碰撞次数和交通违规。与Roach、Autopilot等对比，进行消融实验验证Frenet坐标系和运动学检测的贡献。参数设定如学习率1e-5，训练轮次达数十万次，确保模型收敛。多场景测试验证模型在弯道、密集交通等复杂环境中的表现，确保结果的稳健性和泛化能力。
Results
在CARLA离线排行榜v1中，提出方法提升得分5%，成功率从80%提升到92%；在NoCrash中，成功率由81%提升至95%，碰撞率显著下降。消融实验显示，未引入Frenet坐标系或运动学检测时，性能明显下降，验证了两者的重要性。模型在高曲率道路和复杂交通中表现优异，展现出良好的泛化能力，优于现有控制输出RL模型，验证了其在实际复杂环境中的潜力。
Applications
该架构适用于自动驾驶车辆的路径规划，尤其在城市复杂道路环境中。可作为自主驾驶系统的核心模块，结合感知系统实现端到端的安全导航。未来可扩展到无人驾驶出租车、物流运输等场景，提升路径规划的可解释性与鲁棒性，推动自动驾驶产业的商业化应用。
Limitations & Outlook
模型依赖高质量感知信息，感知误差可能影响路径生成。极端天气或动态障碍物快速变化时，鲁棒性尚待验证。训练成本较高，实际部署需优化算法以满足实时性要求。未来需结合多模态感知与端到端学习，提升系统的适应性与效率。
Plain Language Accessible to non-experts
想象你在开车，路上有很多弯弯绕绕的道路，有时还会遇到交通灯和行人。传统的自动驾驶系统就像是一个非常聪明的司机，它会根据规则直接控制方向盘和油门，但有时候它的决策不够透明，难以理解为什么要这么开。本文提出了一种新方法，就像这个司机提前规划好一条路线，用一条平滑的曲线去引导车辆行驶。这个“路线”是用数学的多项式画出来的，既漂亮又符合车辆的运动规律。通过这种方式，车辆可以更平稳、更安全地行驶，就像有人帮你提前画好一条安全的路径一样。这种方法还能确保路径在车辆的极限范围内，不会出现突然的偏离或失控。它在模拟环境中的表现非常好，成功率高，碰撞少，就像一个经验丰富的司机在复杂的城市道路中游刃有余。未来，这个技术可以让自动驾驶变得更透明、更可靠，就像你有一个懂事又稳重的司机助手一样。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，你可以选择让车自己跑，但如果它只是盲目控制油门和方向盘，你可能不知道它是怎么决定的。现在，想象你让它提前画好一条弯弯曲曲的路线，然后跟着这条路线跑。这就像是在用一条漂亮的曲线引导车子，让它跑得又快又稳。这篇论文就像发明了这样一种“提前画线”的方法，结合了智能学习和数学技巧，确保车子在复杂的城市道路上安全行驶。它会考虑车的极限，比如转弯的角度和速度，确保路径不会太激烈，也不会偏离太远。经过测试，这种方法比以前的更聪明、更安全，能在各种复杂环境中表现得很好。就像你有一个聪明的助手帮你规划路线，让你轻松赢得比赛。这项技术未来可以让自动驾驶变得更像人一样聪明又可靠，不再让人担心它会迷路或出错。
Abstract
Reinforcement learning (RL) has become a prominent framework for developing driving experts in autonomous vehicles. However, most existing RL-based experts are designed to output direct control commands (e.g., throttle, steering), which suffer from a lack of interpretability, high spatial complexity in learning road geometries, and poor compatibility with modern end-to-end planning architectures. To address these limitations, we propose a novel trajectory planning architecture for RL driving experts that integrates an RL policy with a polynomial-based trajectory planner. By employing a Frenet-frame coordinate system, our method simplifies complex road geometries into a curvilinear framework, offering a structured coordinate prior that facilitates policy learning. Furthermore, we incorporate a kinematic feasibility check into the planning stage to ensure that generated trajectories remain within the vehicle's physical limits, effectively mitigating cumulative tracking errors typically found in planning-based systems. We evaluate our approach on key CARLA benchmarks, where it significantly outperforms existing state-of-the-art control-based RL experts. On the CARLA Offline Leaderboard v1 and NoCrash benchmarks, our method improves the driving score by 5% and 11%, respectively, and increases the success rate by 8% and 19%.