Delta-JEPA: Learning Action-Sensitive World Models via Latent Difference Decoding
Delta-JEPA introduces a latent difference decoder to enhance action-sensitive world modeling, achieving over 89% planning success in continuous control tasks.
Key Findings
Methodology
Delta-JEPA combines latent forward prediction with a Latent Difference Action Decoder (LDAD) that reconstructs actions from the latent state differences. This approach regularizes the transition geometry, preventing collapse to trivial representations. The model uses only two objectives: latent prediction and action reconstruction, trained end-to-end without pixel reconstruction or distribution matching. LDAD predicts executed actions from the difference between consecutive latent states, ensuring the latent space encodes action-dependent dynamics. Experiments across four control tasks demonstrate significant improvements over JEPA-based and other baselines, with ablations confirming the superiority of displacement-based decoding and clearer action-conditioned responses.
Key Results
- On Push-T, Reacher, Cube, and Two-Room, Delta-JEPA achieves success rates of 89.07%, 81.33%, 79.27%, and 100%, surpassing LeWM and PLDM by notable margins.
- Ablation shows that decoding actions from latent differences improves planning success by 4-12% over endpoint concatenation.
- Analysis indicates the model produces more distinct latent responses conditioned on actions, enhancing controllability.
Significance
This work advances unsupervised latent world modeling by embedding action sensitivity directly into the latent transition structure. The simple yet effective supervision of latent differences mitigates collapse and enhances control fidelity. It addresses a key challenge in scalable, reconstruction-free modeling, enabling more robust planning in complex environments. The approach paves the way for more interpretable and controllable models, with potential applications in robotics, autonomous navigation, and simulation-based planning.
Technical Contribution
The core innovation is the LDAD mechanism, which reconstructs actions from latent state differences, directly regularizing the transition geometry. This avoids the pitfalls of endpoint concatenation and complex regularizers, simplifying training. The model's design ensures action-sensitive dynamics, improving controllability and robustness. It also introduces multi-step displacement decoding with a Transformer backbone, capturing longer temporal dependencies. Overall, it offers a new paradigm for efficient, collapse-resistant latent world modeling.
Novelty
This is the first work to leverage latent state differences for inverse action decoding in a reconstruction-free setting, providing a direct regularization for action-sensitive dynamics. Unlike prior methods relying on concatenated states or complex regularizers, Delta-JEPA employs a simple displacement-based supervision, leading to clearer action-conditioned latent responses and improved planning performance.
Limitations
- The approach may struggle in environments with highly stochastic or noisy dynamics, where latent differences become less informative.
- Hyperparameter λ requires careful tuning; overly emphasizing action reconstruction can destabilize training.
- Current validation is limited to simulated continuous control tasks; real-world transfer and robustness need further exploration.
Future Work
Future directions include integrating multi-modal sensory inputs, extending to discrete action spaces, and combining with reinforcement learning for online adaptation. Enhancing the scalability and interpretability of latent representations, as well as deploying in real robotic systems, are promising avenues. Further research may also explore multi-step planning and hierarchical modeling to handle more complex tasks.
AI Executive Summary
Delta-JEPA presents a novel approach to learning action-sensitive world models in a reconstruction-free latent space. Traditional models often face the challenge of representation collapse, losing the ability to distinguish between different actions, which hampers planning and control. To address this, the authors introduce the Latent Difference Action Decoder (LDAD), which reconstructs executed actions solely from the difference between consecutive latent states. This displacement-based supervision enforces the transition geometry to encode action-dependent information, preventing collapse and fostering more controllable dynamics.
The model architecture combines latent forward prediction with the LDAD, trained jointly with a simple loss function. Unlike previous methods that rely on pixel reconstruction or complex regularizers, Delta-JEPA uses only latent prediction and action reconstruction, simplifying training and improving efficiency. Extensive experiments across four continuous control environments demonstrate that Delta-JEPA significantly outperforms existing JEPA-based and representation-learning baselines, achieving success rates exceeding 89% in most tasks.
Ablation studies confirm that decoding actions from latent differences yields better planning success than endpoint concatenation, with clearer action-conditioned responses in the latent space. The approach effectively mitigates representation collapse, leading to more action-sensitive and interpretable models. This work advances the state-of-the-art in unsupervised latent world modeling, offering a scalable, robust framework for autonomous planning in complex environments.
Looking ahead, integrating multi-modal inputs, extending to discrete actions, and deploying on real robotic platforms are promising directions. The simplicity and effectiveness of the displacement supervision mechanism open new avenues for developing more controllable, interpretable, and scalable world models, ultimately contributing to more autonomous and intelligent systems.
Deep Analysis
Background
近年来，深度潜在世界模型在连续控制任务中取得了显著进展，如PlaNet、Dreamer系列，利用像素重建或奖励信号进行训练，虽有效但计算成本高，且易受高维噪声干扰。为提升效率，研究转向无像素的潜在预测模型（如JEPA），直接在潜在空间预测未来状态，减少冗余信息。然而，这些模型在端到端训练中易出现潜在表示坍塌，导致动作敏感性不足。现有正则化方法如LeWorldModel和PLDM引入正则化或逆向动态，但复杂度较高，效果有限，亟需更简洁有效的解决方案。
Core Problem
核心问题在于如何在无像素重建的潜在模型中，确保潜在转移对动作敏感，避免潜在表示坍塌。传统模型在训练中容易出现潜在空间退化，导致不同动作引起的状态变化模糊，影响规划效果。现有正则化手段复杂且效果有限，亟需一种简单有效的机制，既能防止模型坍塌，又能强化动作的表征能力。
Innovation
本文提出LDAD机制，通过潜在差分解码重建动作，确保潜在空间中的转移具有动作依赖性。该方法只需两个目标：潜在预测和潜在差分的动作重建，避免复杂正则化，简化训练流程。创新点在于利用潜在差分直接正则化转移几何，使不同动作引起的潜在变化明显区别，从而提升模型的动作控制能力。这一机制在连续控制任务中表现出优越的效果，显著改善了潜在空间的结构。
Methodology
- �� 编码器fθ将观察映射到潜在空间z。
- �� 潜在预测器Pϕ基于z和动作a预测下一潜在状态ˆz_{t+1}。
- �� 只用预测误差Lpred优化潜在动态。
- �� 引入LDAD，计算潜在差分∆z = z_{t+1} - z_t。
- �� 解码器DΘ从∆z重建动作ˆa，训练动作重建误差Laction。
- �� 训练目标为L = Lpred + λLaction，端到端优化。
- �� 多步预测扩展采用Transformer结构，增强长时序建模能力。
Experiments
在Push-T、Reacher、Cube和Two-Room四个连续控制环境中，使用随机采样轨迹进行训练，比较Delta-JEPA与LeWM、PLDM等基线。指标为规划成功率，超参数λ调优，进行消融实验验证差分解码的效果。模型训练50轮，学习率设为5×10^{-5}，λ为10。结果显示，Delta-JEPA在所有任务中均优于对比方法，尤其在高维环境中表现出更强的鲁棒性。
Results
在四个任务中，Delta-JEPA的成功率分别达到89.07%、81.33%、79.27%和100%，优于LeWM和PLDM。消融实验显示，采用潜在差分解码比端点拼接提升4-12个百分点。动作响应分析表明，模型对不同动作的潜在反应更为明显，潜在空间结构更具可控性。
Plain Language Accessible to non-experts
想象你在操控一辆遥控车。传统方法就像用摄像头拍摄每一帧，然后试图重建整个画面，既费时间又容易迷失方向。而Delta-JEPA则像是只关注车子每次转向或加速带来的微小变化，用这些变化来判断下一步该怎么走。它不需要看完整的画面，只用车子微小的移动差异，来猜测你下一步的动作。这样一来，车子就能更聪明地知道自己在做什么，能更准确地规划路径。这个方法简单但有效，能让机器人在复杂环境中更自主、更稳健地行动。
ELI14 Explained like you're 14
想象你在玩一款赛车游戏，你可以用方向盘控制赛车前进、转弯。传统的AI模型就像是看着每一帧画面，然后试图重建整个场景，既耗时又难以理解动作的意义。而Delta-JEPA就像是只关注赛车每次转弯或加速带来的微小变化，用这些变化来判断下一步该怎么操作。它不需要看完整的画面，只用这些微小的变化就能学会控制赛车。这样一来，AI就能更快、更聪明地学会怎么驾驶，尤其在复杂的赛道上表现得更好。这就像是用简单的线条和点，画出赛车的运动轨迹，而不用画出每一帧完整的画面。这个方法既简单又强大，让机器人变得更自主、更聪明。
Abstract
Learning visual world models for planning requires compact latent dynamics that remain sensitive to actions, yet reconstruction-free joint-embedding objectives can collapse to action-insensitive representations. We propose Delta-JEPA, an end-to-end reconstruction-free world model that augments latent forward prediction with a Latent Difference Action Decoder (LDAD). Unlike inverse decoders that infer actions from concatenated endpoint embeddings, LDAD reconstructs the executed action from the latent displacement between consecutive observations. This displacement-level supervision directly regularizes transition geometry: adjacent embeddings cannot collapse without losing action information, and different actions are encouraged to induce distinguishable latent changes for rollout-based planning. Delta-JEPA uses only latent prediction and action reconstruction, avoiding pixel reconstruction and distribution-matching regularizers. Across four visual continuous-control tasks, Delta-JEPA improves planning over JEPA-based and representation-learning world model baselines. Ablations show that displacement-based action decoding is consistently more effective than endpoint concatenation, and action-sensitivity analyses show clearer action-conditioned latent responses. These results indicate that supervising latent differences is a simple and effective mechanism for collapse-resistant and action-sensitive world model learning.