ActionSplice: In-Flight Action Editing for Interactive World Models
ActionSplice uses Counterfactual State Transport to enable in-flight action editing, reducing LPIPS by up to 75.9% and speeding up sampling 2.73×.
Key Findings
Methodology
This paper introduces ActionSplice, which leverages Counterfactual State Transport (CST) by training residual predictors to correct interrupted backbone-native states during sampling. It features two variants: CSTR (full chunk retargeting) and CSTT (temporal prefix-preserving splicing). The approach uses matched rollback supervision to generate training targets, enabling the model to learn state corrections without re-evaluating completed steps. During inference, the frozen world model and sampler are quickly adjusted via learned residuals, allowing seamless in-flight action updates. Extensive experiments on minWM-Wan Action2V and HY-WM1.5 demonstrate significant reductions in LPIPS (up to 75.9%) and substantial speedups (up to 2.73×), outperforming traditional condition swapping and partial rollback methods.
Key Results
- On minWM-Wan Action2V, CSTR reduces rollback-relative LPIPS by 61.5%, PSNR increases by 3.33 dB, and boundary error decreases. On HY-WM1.5, LPIPS drops by 75.9%, PSNR improves by 5.31 dB, with 1.57× faster response than waiting. CSTT achieves suffix LPIPS reductions of 56.1% and 77.5%, with 2.73× and 1.69× speedups, respectively.
- The method effectively handles in-sampling action changes, maintaining high visual fidelity while drastically reducing latency. It generalizes across different backbone architectures, showing robustness in diverse scenarios.
Significance
This work addresses a critical bottleneck in interactive video world models: enabling real-time, in-flight action edits without sacrificing quality or speed. It introduces a novel state correction paradigm that circumvents the need for costly re-evaluation, thus paving the way for more responsive virtual environments, gaming, and simulation systems. The approach enhances user experience by reducing control latency and increasing flexibility, marking a significant step forward in dynamic scene editing.
Technical Contribution
The core innovation lies in the integration of matched rollback supervision with residual learning to perform in-flight state corrections. Unlike prior methods relying on shared latent spaces or weight transfer, ActionSplice operates directly on backbone-native states, enabling model-agnostic, efficient corrections. The dual variants (CSTR and CSTT) provide flexible control granularity. The framework also introduces a training protocol that pairs interrupted states with their matched rollback targets, ensuring high-fidelity correction learning. This design significantly reduces the need for re-evaluation, offering a new paradigm for interactive video synthesis.
Novelty
This is the first system to perform in-sampling action editing by learning state corrections directly in frozen models, leveraging matched rollback supervision. Its key innovation is the residual-based correction mechanism that avoids re-evaluation, unlike traditional condition swapping or full rollback. The dual variants for different temporal granularities provide versatile control, making this approach uniquely suited for real-time applications. This represents a new direction in interactive, controllable video generation.
Limitations
- The reliance on matched rollback training samples may limit performance in highly complex or unseen scenarios. The residual predictor's accuracy diminishes with increasing scene complexity, potentially affecting fidelity. Repeated interruptions could lead to cumulative state drift, requiring further robustness improvements.
Future Work
Future research will focus on enhancing residual prediction robustness, integrating multi-modal cues for semantic consistency, and extending the framework to multi-agent or multi-modal scenarios. Additionally, optimizing computational efficiency and exploring unsupervised training methods could further improve scalability and applicability in real-world systems.
AI Executive Summary
Interactive control over video world models is essential for immersive virtual environments, gaming, and simulation. However, existing models often struggle with real-time responsiveness when abrupt action changes occur during sampling. Traditional solutions, such as condition swapping or full re-evaluation, either introduce significant latency or compromise visual fidelity. This gap limits the deployment of highly interactive systems where immediate action updates are crucial.
This paper introduces ActionSplice, a novel framework that enables in-flight action editing through Counterfactual State Transport (CST). The key idea is to learn how to quickly adjust the internal state of a frozen world model to reflect new actions without re-running previous evaluations. By training residual predictors on matched rollback pairs, ActionSplice can transport interrupted states toward their counterfactual targets, ensuring the updated action is incorporated seamlessly. The framework features two variants: CSTR, which updates the entire active chunk, and CSTT, which preserves the temporal prefix and updates only the suffix, providing fine-grained control.
Extensive experiments on two benchmarks, minWM-Wan Action2V and HY-WM1.5, demonstrate the effectiveness of ActionSplice. Results show up to 75.9% reduction in LPIPS error, PSNR improvements of over 3 dB, and speedups exceeding 2.7× compared to waiting. These improvements translate into more responsive, higher-quality video generation, crucial for real-time applications. The approach also outperforms traditional methods like condition swapping and partial rollback, highlighting its robustness and efficiency.
The significance of this work lies in its ability to fundamentally change how interactive video models handle in-sampling action changes. By eliminating the need for costly re-evaluation and enabling instant state corrections, it opens new possibilities for immersive VR, gaming, and dynamic scene editing. Despite its success, challenges remain in scaling to more complex scenes and multiple interruptions, motivating future research into more robust residual learning and multi-modal integration. Overall, ActionSplice marks a major step toward truly responsive and controllable virtual environments.
Deep Analysis
Background
近年来，视频世界模型的发展极大推动了虚拟环境、游戏和仿真技术的进步。基于扩散模型、变分自编码器和自回归机制的系统已实现高质量的连续视频生成，代表性工作包括Valevski等的扩散模型、Wang等的多段自回归模型，以及HunyuanWorld的实时生成系统。这些模型在保持几何一致性和长序列连续性方面取得了突破，但在应对突发动作变化、实时交互和状态修正方面仍存在瓶颈。传统方法多依赖等待下一帧或重跑全部评估，导致响应延迟，限制了其在动态交互场景中的应用。
Core Problem
核心难题在于：在采样过程中，用户突发动作变更时，如何快速修正模型状态以反映新控制，而不牺牲生成连续性和效率。现有方案如条件交换和全重跑，存在状态偏差大、响应慢的问题。尤其在交互式应用中，用户希望能在不中断生成流程的情况下，实时调整动作参数。这对模型的中途修正能力提出了极高的要求，限制了虚拟环境的交互性和灵活性。
Innovation
本文提出的ActionSplice创新性地结合匹配回滚和残差学习，解决了在冻结状态下实现飞行中动作编辑的难题。具体创新点包括：1）利用匹配回滚生成目标状态，提供监督信号；2）设计两类修正策略（全块和时序局部），满足不同粒度控制需求；3）在模型冻结的前提下，学习残差迁移，实现快速状态修正；4）避免重跑已完成的评估，极大提升响应速度。这些创新突破了传统的限制，为实时视频交互提供了新思路。
Methodology
- �� 训练残差预测器：利用匹配回滚对中断状态，学习在冻结模型状态下的状态迁移残差；• 设计掩码机制：限制修正范围，只在可编辑区域调整状态；• 生成监督目标：通过匹配回滚得到目标状态，训练残差预测器；• 采样修正：在中断点利用预测残差，快速修正状态，继续采样；• 两类变体：CSTR支持全块更新，CSTT支持局部（时序）更新，满足不同控制粒度；• 训练策略：利用匹配回滚生成训练样本，确保模型在不同场景下的泛化能力。
Experiments
在minWM-Wan Action2V和HY-WM1.5两个基准上，采用150个场景提示，进行多轮中断采样训练。对比方法包括等待、条件交换、部分重跑、重噪声和理想全回滚。评估指标涵盖LPIPS、PSNR、SSIM、边界误差和响应延迟。通过不同中断点和控制边界，验证ActionSplice在保持生成质量的同时，显著提升响应速度。实验还包括ablation研究，分析不同变体和掩码策略的影响。
Results
在LPIPS指标上，CSTR较直接条件交换降低61.5%（minWM）和75.9%（HY-WM1.5）；CSTT在suffix LPIPS上分别降低56.1%和77.5%。速度方面，CSTT比等待快2.73倍（minWM）和1.69倍（HY-WM1.5）。在图像质量方面，CSTR达到25.66 dB的PSNR，0.6902的SSIM，LPIPS为0.1337，优于对比方法。整体结果显示，该方法在保持高质量生成的同时，大幅提升交互响应速度。
Applications
该技术适用于虚拟现实、游戏引擎、交互式仿真等场景，支持用户在不中断生成流程的情况下，实时调整动作参数。只需少量训练样本即可实现快速修正，极大缩短了响应延迟。未来可结合多模态信息，增强场景理解和动作语义一致性，推动沉浸式虚拟环境的发展。
Limitations & Outlook
目前方法依赖匹配回滚训练样本，难以应对极端复杂场景或连续多次中断带来的偏差累积。残差预测在高复杂度场景下仍存在偏差，可能影响生成质量。此外，模型在长序列或多次中断后，状态偏差可能逐渐放大，未来需引入更鲁棒的状态校正机制和多模态信息融合以提升性能。
Plain Language Accessible to non-experts
想象你在厨房做菜，突然有人打断你，要求你换个菜谱。传统做法是重新开始，浪费时间；或者只调整部分步骤，但可能做得不自然。ActionSplice就像有个聪明的助手，能在你还在炒菜时，偷偷观察到你被打断了，然后根据你想换的菜谱，快速调整锅里的内容，继续炒菜，而不用重新开始。它学会了在你还没完成的部分偷偷补救，确保菜肴看起来还很美味。这就像厨房里的魔法师，能在你忙碌中帮你快速改菜，既省时间又保证效果。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，突然想换个角色或者技能，但你还在战斗中。以前的方法是暂停游戏，重新加载新角色，花很多时间。现在，有个聪明的助手可以在你还在战斗的时候，偷偷帮你调整角色的状态，让你不用暂停，直接继续战斗。这就像你有个隐形的帮手，能在你不注意时，把你的角色状态悄悄改好，然后你还能继续打怪，不卡顿。这种技术让游戏变得更流畅，反应更快，玩家体验也更棒。
Glossary
Counterfactual State Transport (CST)（反事实状态迁移）
一种在冻结模型状态下，通过学习残差实现状态快速修正的方法。技术上利用匹配回滚生成目标状态，避免重跑已完成的评估。
本文中用于实现飞行中动作编辑的核心机制。
匹配回滚（Matched Rollback）
通过重放部分评估，生成与中断状态对应的目标状态，用于训练残差预测器。确保修正的状态与理想状态一致。
训练中用来提供监督信号。
LPIPS指标
衡量生成内容视觉相似度的指标，越低表示图像越相似。用于评估生成视频的质量。
评估生成内容的视觉一致性。
PSNR（Peak Signal-to-Noise Ratio）
衡量图像或视频重建质量的指标，数值越高代表质量越好。
用于比较生成内容的清晰度。
Open Questions Unanswered questions from this research
- 1 如何进一步提升残差预测在极端复杂场景中的准确性，特别是在多次中断和连续动作变换情况下的鲁棒性。
- 2 是否可以结合多模态信息（如语音、文本）实现更智能的动作修正和场景理解。
Applications
Immediate Applications
虚拟现实交互控制
支持用户在虚拟环境中实时调整动作，无需等待下一帧，提升沉浸感和交互体验。
游戏中的动态场景编辑
允许玩家在游戏中即时改变角色动作或场景状态，增强游戏的流畅性和趣味性。
Long-term Vision
智能虚拟助手
未来可以实现自主学习和修正，支持复杂场景中的多模态交互，推动虚拟环境的智能化。
Abstract
Chunk-autoregressive video world models typically condition each generated chunk on one action. An action received during sampling must therefore wait for the next chunk, condition future solver evaluations on a state produced under the previous action, or trigger rollback that repeats completed evaluations. We introduce ActionSplice, an inference framework that formulates this problem as Counterfactual State Transport (CST). A lightweight corrector transports the interrupted backbone-native representation toward the matched state induced by the revised action at the same solver step. The world model and sampler remain frozen, and sampling resumes without replaying completed evaluations. The retargeting variant $\mathrm{CST}*{R}$ updates the entire active chunk, while the temporal-splicing variant $\mathrm{CST}*{T}$ preserves a temporal prefix and updates only the suffix. Across minWM-Wan Action2V and HY-WM1.5, $\mathrm{CST}*{R}$ reduces rollback-relative LPIPS by 61.5% and 75.9% relative to direct condition swapping. $\mathrm{CST}*{T}$ reduces suffix LPIPS by 56.1% and 77.5%, respectively, while providing $2.73\times$ and $1.69\times$ pixel-ready speedups over waiting. Under the HY-WorldPlay protocol, $\mathrm{CST}_{R}$ obtains a PSNR of 25.66 dB, an SSIM of 0.6902, and an LPIPS of 0.1337 against the original rollout.