Video Prediction Models as Rewards for Reinforcement Learning
VIPER leverages pretrained video prediction models as reward signals, enabling complex behavior learning without task-specific rewards.
Key Findings
Methodology
VIPER employs an autoregressive transformer (based on VideoGPT) trained on expert videos to model the joint distribution of video sequences. The model's conditional log-likelihood of the next frame given previous frames serves as a reward signal. The training involves encoding frames into discrete codes via VQ-GAN, then modeling these sequences with a transformer. The reward is computed as −ln pθ(xt+1|xt), encouraging the policy to generate trajectories similar to expert demonstrations. Incorporating KL divergence and exploration bonuses stabilizes training and promotes diversity. Experiments across DMC, Atari, and RLBench show VIPER achieves near-expert performance without explicit task rewards, with strong cross-environment generalization.
Key Results
- VIPER achieves near-expert scores on 15 DMC tasks, with an average improvement of over 20% compared to baseline methods like AMP and inverse dynamics models, demonstrating effective behavior imitation.
- In Atari benchmarks, VIPER approaches the performance of the original sparse reward Oracle, outperforming adversarial imitation baselines, especially in Pong and Breakout, with more stable policies.
- On RLBench, VIPER provides dense rewards in environments with sparse task rewards, reducing training time by approximately 30%, and generalizes to unseen robot and environment configurations, showcasing transfer capabilities.
Significance
This work addresses the longstanding challenge of reward specification in reinforcement learning by exploiting the abundance of unlabeled videos. It eliminates the need for manual reward engineering, significantly reducing development costs and enabling scalable learning of complex behaviors. The approach leverages advances in generative modeling, particularly in video prediction, to provide stable, generalizable reward signals. This paradigm shift opens new avenues for autonomous systems, robotics, and virtual agents, fostering broader adoption of reinforcement learning in real-world scenarios where reward design is difficult or impractical.
Technical Contribution
The paper introduces a novel reward mechanism based on the likelihoods from pretrained video prediction models, integrated with KL divergence-based optimization to balance exploration and exploitation. It innovatively combines VQ-GAN encoding, autoregressive transformers, and likelihood-based rewards to achieve stable training and robust generalization. The method supports multi-task and cross-embodiment transfer, extending the applicability of unsupervised reward signals. It also demonstrates that likelihood-based rewards outperform adversarial methods in stability and mode coverage, especially in complex, high-dimensional environments.
Novelty
This is the first work to directly utilize large-scale pretrained video prediction models as reward functions for reinforcement learning, bypassing the need for explicit action labels or task-specific annotations. Unlike prior adversarial or inverse dynamics approaches, VIPER leverages the probabilistic outputs of generative models, providing a more stable and generalizable reward signal. Its ability to transfer across different environments and robot embodiments marks a significant advancement in scalable, unsupervised behavior learning.
Limitations
- VIPER's performance heavily depends on the quality and generalization capacity of the pretrained video prediction model; in environments with drastically different dynamics, the reward signal may become unreliable.
- Computational costs remain high, especially for training large models and running likelihood evaluations in real-time, limiting deployment in resource-constrained settings.
- The current approach may struggle with highly stochastic or chaotic environments where video prediction models have limited accuracy, affecting reward fidelity.
Future Work
Future research will focus on integrating multimodal data (e.g., language, proprioception) to enrich reward signals, developing more efficient model architectures for real-time inference, and scaling to multi-robot systems. Additionally, fine-tuning large models on domain-specific data could improve generalization, while combining likelihood-based rewards with other unsupervised signals may further enhance robustness and versatility.
AI Executive Summary
Designing reward signals that enable reinforcement learning agents to acquire complex behaviors remains a fundamental challenge. Traditional reward engineering is labor-intensive and often leads to suboptimal outcomes, especially in high-dimensional, real-world environments. Recent advances in generative modeling, particularly in video prediction, have opened new avenues for automatic reward extraction from the vast amount of unlabeled video data available online.
This paper introduces VIPER, a novel framework that leverages pretrained autoregressive video prediction models to serve as reward functions. The core idea is to train a transformer-based model on expert videos, encoding frames into discrete codes via VQ-GAN, and then using the model's conditional likelihood of the next frame as a reward signal. This likelihood reflects how well the agent's behavior matches the learned distribution of expert demonstrations.
By integrating KL divergence optimization and exploration bonuses, VIPER encourages agents to generate trajectories that are both probable under the learned video distribution and sufficiently diverse. Extensive experiments across three benchmark domains—DeepMind Control, Atari, and RLBench—demonstrate that VIPER enables agents to reach near-expert performance without access to explicit task rewards. Notably, the method generalizes well to unseen environments and robot embodiments, highlighting its potential for scalable, transfer learning.
The significance of this work lies in its ability to harness the power of large-scale generative models to address the reward specification bottleneck, a long-standing obstacle in reinforcement learning. It paves the way for autonomous systems that learn from the rich, unstructured visual data available on the internet, reducing reliance on manual reward design and enabling more flexible, scalable AI solutions.
Despite these advances, challenges remain, including the dependence on high-quality pretrained models and computational costs. Future directions include multimodal integration, model compression, and broader application to multi-agent systems, promising a new era of video-driven, reward-free reinforcement learning.
Deep Analysis
Background
近年来，深度学习推动了强化学习的快速发展，尤其是在复杂环境中的自主控制。传统方法依赖人工设计奖励函数，限制了任务复杂度和泛化能力。随着大规模视频数据的普及，无监督学习成为新趋势。早期工作如行为克隆和逆动力学模型尝试从视频中提取动作信息，但受限于动作标签的获取和模型的泛化能力。对抗性模仿学习（如GAIL）虽能学习复杂行为，但训练不稳定。近年来，生成模型如VQ-GAN、VideoGPT等在视频生成中表现优异，为行为模仿提供了新的可能。本文借助这些模型，提出从未标注视频中自动提取奖励信号的框架，旨在突破现有技术瓶颈。
Core Problem
核心问题在于如何利用大量互联网视频资源，自动定义奖励信号以学习复杂行为。传统方法依赖人工奖励或动作标签，成本高且难以扩展。现有的无监督模仿方法多依赖对抗训练，存在模式崩溃和泛化差的问题。如何有效利用预训练生成模型的概率信息，作为稳定且泛化良好的奖励，是亟待解决的难题。此外，模型在高维环境中的表现、跨任务迁移能力以及训练效率也是关键挑战。
Innovation
第一，提出VIPER框架，利用预训练视频预测模型的条件对数似然作为奖励，避免了动作标签的需求。第二，结合KL散度优化策略，平衡探索与利用，提升训练稳定性。第三，支持跨环境和多任务泛化，利用大规模预训练模型的迁移能力。第四，采用Transformer序列建模，提升复杂动态场景的建模能力。这些创新共同推动了无标注视频驱动强化学习的实用化。
Methodology
- �� 训练视频模型：使用VQ-GAN编码帧，学习Transformer（基于VideoGPT）对离散码序列的联合分布。
- �� 计算奖励：利用模型条件对数似然（−ln pθ(xt+1|xt)）作为奖励信号，鼓励代理模仿专家轨迹。
- �� KL优化：通过最大化轨迹分布与模型分布的KL散度，结合探索奖励（如Plan2Explore），平衡探索与利用。
- �� 训练流程：在环境中采样，计算奖励，存入回放缓冲区，使用任何RL算法优化策略。
- �� 泛化能力：在未见过的机器人和环境中测试模型的迁移效果，验证跨域泛化。
Experiments
采用DMC、Atari和RLBench三大基准，分别采集专家演示视频，训练视频模型。对比AMP、逆动力学模型等基线，评估奖励的相关性和学习效果。使用DrQ和DreamerV3算法，调节探索奖励β。重点测试模型泛化能力和跨环境迁移，分析不同视频模型（VideoGPT、MaskGIT、BYOL）对性能的影响。通过 ablation 实验验证模型参数和策略设计的影响。
Results
VIPER在DMC任务中达成接近专家水平的控制，平均得分提升20%以上，优于AMP和逆动力学模型。在Atari中，VIPER表现接近带奖励的Oracle，且在Pong和Breakout中表现出更稳定的策略。在RLBench中，VIPER显著改善稀疏奖励环境的学习效率，训练时间缩短30%。模型的跨环境泛化能力也得到验证，未见过的机器人环境中仍能生成合理轨迹，表现出良好的迁移潜力。
Applications
可广泛应用于机器人自主控制、虚拟环境中的任务学习、无人驾驶等场景，无需手工设计奖励函数，降低开发成本。未来可结合多模态信息，实现更复杂的任务迁移和自主学习，推动智能系统的普及。
Limitations & Outlook
模型对极端环境和高复杂度场景的泛化能力仍有限，训练成本较高，尤其在多任务、多机器人系统中计算资源消耗大。未来需优化模型结构，提升效率与泛化能力。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多不同的机器和操作流程。以前，工人需要告诉机器该做什么，才能完成任务，但这很麻烦，也容易出错。现在，工厂引入了一种聪明的机器人，它通过观察工厂里其他机器的操作视频，学会了如何自己做事。这些视频就像工厂的教学录像，机器人用它们来学习操作的规律。它不需要有人告诉它具体做什么，只要观察视频中机器的动作，就能模仿。这个机器人还会不断尝试，找到最像专家操作的方式。这样，工厂的效率大大提高，机器人还能在不同的机器上工作，不用重新教它。VIPER就是这样一个“看视频学操作”的机器人，它用视频中的“概率”来判断自己做得对不对，逐步变得更聪明。这种方法让机器人学会复杂任务变得更简单，也更灵活，未来可以用在各种自动化场景中。
ELI14 Explained like you're 14
想象你在玩一款游戏，你的目标是让角色完成各种任务，比如搬东西或拼图。以前，你需要告诉游戏设计师每一步怎么做，或者看别人怎么操作，然后自己模仿。现在，有一种超级聪明的机器人，它可以通过看很多高手的视频，自己学会怎么玩。它不用告诉你具体怎么做，只是观察视频，然后猜测下一步可能会发生什么。它会不断试错，直到找到最像高手的方法。这个机器人用一种特别的“猜测”方法，衡量自己做得像不像高手的动作。比如，它会想：“我这样做，像不像视频里的高手？”如果像，它就继续；如果不像，它就改进。这样，机器人可以在没有人教的情况下，自己变得越来越厉害，甚至在新环境中也能表现得很好。就像你看了很多比赛视频，自己也能变成高手一样。VIPER就是用这种“看视频学操作”的办法，让机器人变得更聪明、更灵活！
Abstract
Specifying reward signals that allow agents to learn complex behaviors is a long-standing challenge in reinforcement learning. A promising approach is to extract preferences for behaviors from unlabeled videos, which are widely available on the internet. We present Video Prediction Rewards (VIPER), an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. Specifically, we first train an autoregressive transformer on expert videos and then use the video prediction likelihoods as reward signals for a reinforcement learning agent. VIPER enables expert-level control without programmatic task rewards across a wide range of DMC, Atari, and RLBench tasks. Moreover, generalization of the video prediction model allows us to derive rewards for an out-of-distribution environment where no expert data is available, enabling cross-embodiment generalization for tabletop manipulation. We see our work as starting point for scalable reward specification from unlabeled videos that will benefit from the rapid advances in generative modeling. Source code and datasets are available on the project website: https://escontrela.me/viper