GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving
Proposed GRI combines offline expert demonstrations with online exploration, boosting urban autonomous driving scores by 17%.
Key Findings
Methodology
The GRI framework integrates offline expert demonstrations with online RL exploration by treating demonstrations as perfect behaviors with fixed high rewards. An offline demonstration agent feeds expert data into the replay buffer alongside exploration data, enabling seamless joint training across any off-policy RL algorithm. Visual encoders pretrained on auxiliary tasks extract semantic features from multi-camera inputs, which are then frozen during RL training. The decision network, trained with Rainbow-IQN Ape-X, discretizes actions for urban driving. Experiments show GRI significantly outperforms prior methods, achieving a 17% improvement on CARLA leaderboard, with only three cameras and no LiDAR, demonstrating high efficiency and robustness.
Key Results
- On CARLA leaderboard, GRI achieved a score of 61.85, surpassing the previous top method World on Rails at 54.5, with fewer sensors, indicating superior urban driving performance.
- In NoCrash benchmark, GRI reached 98.3% route completion in Town01 after 12M steps, outperforming pure RL's 94%, showing better generalization.
- In MuJoCo tasks, with 20% demonstration data, models trained with GRI outperformed vanilla RL, converging faster and achieving higher rewards across environments like HalfCheetah-v2 and Humanoid-v2.
Significance
This work addresses the sample inefficiency and stability issues of DRL in complex environments by effectively leveraging expert demonstrations. It advances autonomous driving by reducing data requirements and improving safety, robustness, and generalization. The framework's versatility extends to robotics and decision-making domains, offering a practical pathway toward real-world deployment with less reliance on extensive labeled data.
Technical Contribution
The core innovation lies in the GRI framework, which simplifies integration with any off-policy RL algorithm via the offline demonstration agent. By assigning a fixed high reward to expert data, it enables stable, efficient learning. The multi-camera visual encoding enhances perception, while the seamless fusion of demonstration and exploration data accelerates training and improves robustness. The approach's simplicity and effectiveness set it apart from prior complex methods.
Novelty
This is the first systematic approach to treat expert demonstrations as perfect data with fixed high rewards, integrating them into off-policy RL via an offline agent. Unlike SQIL and DQfD, GRI does not rely on environment rewards for demonstrations, simplifying training and broadening applicability. Its unified framework effectively combines imitation and exploration, marking a significant step forward.
Limitations
- The assumption that expert demonstrations are perfect may not hold in real-world noisy data, potentially affecting performance.
- Dependence on multi-camera setups increases hardware complexity and cost.
- Generalization to highly dynamic or unseen environments still requires further validation.
Future Work
Future research will explore adaptive reward schemes to handle noisy demonstrations, multi-source demonstration fusion, and transfer to real-world autonomous vehicles. Enhancing robustness and reducing sensor dependency are key directions to facilitate commercial deployment.
AI Executive Summary
Autonomous driving remains a formidable challenge due to the environment's complexity and the high data demands of deep reinforcement learning (DRL). While DRL offers adaptive decision-making, its sample inefficiency and instability hinder real-world deployment. Imitation learning (IL), leveraging expert demonstrations, provides a faster alternative but suffers from distribution mismatch. To bridge these gaps, this paper introduces the General Reinforced Imitation (GRI) framework, a novel approach that combines offline expert demonstrations with online exploration seamlessly.
The core idea is to treat expert data as perfect behavior, assigning a fixed high reward, and incorporate it into the experience replay buffer alongside exploration data. This is achieved through an offline demonstration agent that feeds expert samples into the training process, compatible with any off-policy RL algorithm such as SAC or Rainbow-IQN. The visual perception module employs pretrained EfficientNet encoders on auxiliary tasks like semantic segmentation and classification, extracting rich environment features from multi-camera inputs, which are then frozen during RL training.
Experiments conducted in the CARLA simulator demonstrate that GRI significantly outperforms prior state-of-the-art methods, achieving a 17% higher driving score on the leaderboard. Notably, it accomplishes this with only three cameras and no LiDAR sensors, highlighting its efficiency. The approach also generalizes well to MuJoCo continuous control tasks, where incorporating 20% demonstration data accelerates learning and improves robustness.
This research advances autonomous driving by reducing data requirements and improving safety and robustness. Its simplicity, broad applicability, and strong empirical results suggest it can accelerate real-world deployment and inspire future work on combining imitation and reinforcement learning in complex environments. Limitations include assumptions of perfect demonstrations and hardware costs, but ongoing work aims to address these challenges, pushing toward more practical, scalable autonomous systems.
Deep Analysis
Background
近年来，深度强化学习（DRL）在自主驾驶和机器人控制中取得显著进展，代表算法如DQN、SAC、DDPG等不断优化样本利用率。模仿学习（IL）通过专家演示快速学习行为，但受限于分布偏差。结合两者的混合方法逐渐成为研究热点，诸如DQfD和SQIL等尝试利用演示数据提升效率。城市自动驾驶场景复杂，环境多变，单一方法难以兼顾样本效率与鲁棒性。近年来，基于视觉感知的端到端系统成为研究焦点，CARLA模拟器提供了理想平台。尽管如此，如何高效融合专家知识与探索经验，仍是亟待解决的问题。
Core Problem
核心问题在于深度强化学习在复杂环境中训练成本高、样本需求大，且易受环境分布偏差影响。模仿学习虽能快速学习，但在偏离演示分布时表现不佳。现有方法如SQIL和DQfD存在利用效率不足或依赖环境奖励的问题。如何设计一种既能充分利用专家演示，又能保持探索能力的通用框架，成为提升自主驾驶系统性能的关键。特别是在视觉感知场景中，如何融合多摄像头信息、提升感知鲁棒性，也是亟待攻克的难题。
Innovation
本文提出的GRI方法创新点在于：1）将专家演示视为完美行为，赋予固定高奖励，简化奖励设计；2）引入离线演示代理，将专家数据与探索数据同时加入经验回放，实现两者的无缝融合；3）适配任何离策略强化学习算法，具有良好的通用性；4）结合多摄像头视觉编码器，提升感知能力。该框架在保持算法简洁的同时，显著提升样本效率和泛化能力，突破了传统方法的局限。
Methodology
- �� 视觉编码器预训练：利用大规模标注数据，训练两个Efficientnet-b1模型，进行语义分割和分类任务，提取环境丰富信息。• 冻结编码器：在强化学习训练中，保持编码器参数不变，作为特征提取器。• 训练决策网络：基于编码器输出，使用Rainbow-IQN Ape-X算法，离散动作空间（27个转向值×4个油门/刹车值）进行训练。• 结合演示：离线演示代理将专家数据（含状态、动作、固定高奖励）同步加入经验池，探索代理进行环境探索，二者数据共同训练。• 训练流程：在每个训练步骤，按概率选择来自演示或探索的episode，确保专家知识的持续引入。• 视觉信息融合：多摄像头图像经过编码后拼接，增强环境感知。• 训练目标：最大化累积奖励，兼顾探索与模仿。
Experiments
在CARLA平台，训练60M步（含200,000专家演示样本）进行城市驾驶任务，比较GRI与纯RL、IL方法。在MuJoCo环境，采用不同算法（SAC、DDPG）验证泛化能力。指标包括驾驶得分、道路完成率和碰撞次数。通过不同演示比例（如20%、50%）进行消融分析，评估样本效率和训练速度。实验还测试噪声演示对性能的影响，验证框架鲁棒性。
Results
GRI在CARLA排行榜中，驾驶得分达61.85，超越前一代方法World on Rails的54.5，提升17%。在NoCrash测试中，Town01环境下，测试道路完成率达98.3%，优于纯RL的94%，显示出更强的泛化能力。在MuJoCo环境中，加入20%演示比例的模型在HalfCheetah-v2和Humanoid-v2中表现优异，训练速度快，泛化能力强。整体结果显示，融合专家演示显著提升样本效率和泛化能力，且在复杂环境中表现优越。
Applications
该方法适用于自动驾驶、机器人控制、复杂决策系统等场景，尤其在数据有限或环境变化频繁时表现出色。只需少量专家演示，即可快速训练出鲁棒策略，减少对大量标注数据的依赖。未来，结合多源演示和真实车辆系统，将推动自动驾驶的商业化部署，降低成本，提高安全性。
Limitations & Outlook
假设专家演示为完美行为，可能在实际中存在噪声或偏差，影响效果。对多摄像头配置依赖较大，硬件成本较高。在极端复杂环境或未见场景中，泛化能力仍需验证。未来需解决演示数据质量不稳定和多源融合问题。
Plain Language Accessible to non-experts
想象你在厨房做饭，老师（专家）给你一份完美的食谱（演示），告诉你每一步怎么做。你自己也尝试探索不同的做法（探索），但有时候会偏离食谱，做出不太好吃的菜。GRI就像一个聪明的厨师，把老师的完美食谱作为一个高分奖励，和自己探索的经验一起存放在厨房的记事本里。这样，厨师既能学习老师的好方法，又能通过自己尝试改进。最终，他能在没有老师指导的情况下，也做出美味的菜。这种方法让厨师学得快，做得好，也能应对不同的厨房环境，就像自动驾驶系统在复杂城市中学会安全驾驶一样。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，里面有很多不同的关卡和挑战。你可以跟高手（专家）学习他们的技巧，也可以自己试着探索不同的策略。可是，光靠模仿高手，可能会遇到一些你不懂的难题；只靠自己探索，又可能走弯路很慢。GRI就像是一个聪明的游戏助手，它把高手的最佳操作（演示）作为奖励，和你自己探索的经验一起存到一个记忆箱里。每次你遇到新关卡时，它会告诉你高手的操作，让你少走弯路，也能更快学会怎么赢。这样，你既能学会高手的绝招，又能自己探索出新玩法，变得越来越厉害。这就像让自动驾驶汽车既能模仿专家的驾驶，又能自己学会应对各种复杂路况一样。
Glossary
Deep Reinforcement Learning (深度强化学习)
一种通过深度神经网络实现的强化学习方法，用于智能体在复杂环境中自主学习最优策略。
本文采用DRL训练自主驾驶模型。
Imitation Learning (模仿学习)
让模型模仿专家行为，从演示数据中学习策略，减少探索成本。
作为GRI的基础之一。
Experience Replay (经验回放)
存储训练样本的缓冲区，供强化学习算法采样训练。
核心机制之一。
Off-policy RL (离策略强化学习)
允许在训练过程中使用不同策略生成的数据，增强学习效率。
GRI兼容的算法类型。
CARLA
一个开源的自动驾驶模拟器，用于研究和验证自主驾驶算法。
实验平台之一。
Open Questions Unanswered questions from this research
- 1 如何在真实环境中应对专家演示中的噪声和偏差，仍是挑战。未来需研究鲁棒性增强机制，以确保在实际应用中效果稳定。
Applications
Immediate Applications
城市自动驾驶系统
利用GRI快速训练高鲁棒性驾驶策略，减少标注成本，提升安全性。
机器人自主控制
在复杂环境中融合专家演示，实现高效自主导航。
Long-term Vision
智能交通管理
实现全自动化、智能化的交通调度与控制系统，减少交通事故和拥堵。
Abstract
Deep reinforcement learning (DRL) has been demonstrated to be effective for several complex decision-making applications such as autonomous driving and robotics. However, DRL is notoriously limited by its high sample complexity and its lack of stability. Prior knowledge, e.g. as expert demonstrations, is often available but challenging to leverage to mitigate these issues. In this paper, we propose General Reinforced Imitation (GRI), a novel method which combines benefits from exploration and expert data and is straightforward to implement over any off-policy RL algorithm. We make one simplifying hypothesis: expert demonstrations can be seen as perfect data whose underlying policy gets a constant high reward. Based on this assumption, GRI introduces the notion of offline demonstration agents. This agent sends expert data which are processed both concurrently and indistinguishably with the experiences coming from the online RL exploration agent. We show that our approach enables major improvements on vision-based autonomous driving in urban environments. We further validate the GRI method on Mujoco continuous control tasks with different off-policy RL algorithms. Our method ranked first on the CARLA Leaderboard and outperforms World on Rails, the previous state-of-the-art, by 17%.