WorldKV: Efficient World Memory with World Retrieval and Compression

Autoregressive video diffusion models have emerged as a powerful paradigm for real-time, action-conditioned video generation. By sequentially generating frames conditioned on previous frames and actions, these models enable interactive applications such as gaming, embodied AI agents, and robotic simulation. Representative works include LingBot-World, which scales to minute-level videos with full KV-cache attention, and Matrix-Game-2.0, which operates on shorter sequences with sliding-window inference. Despite advances in generation quality, maintaining long-term spatial and temporal consistency—critical for persistent, explorable worlds—remains challenging. Full-history KV-cache attention preserves consistency by attending to all past latent states but incurs linearly growing memory and computational costs, limiting real-time feasibility. Sliding-window inference bounds these costs but sacrifices access to long-term context, causing hallucination and drift. Prior solutions involve external memory banks, learnable compression, or explicit 3D scene representations, which require additional training or incur inference latency. This paper builds on the insight that the model's own KV cache functions as an emergent memory, proposing a training-free framework to efficiently manage this cache for long-term consistency.

The core problem addressed is how to enable persistent, consistent world memory in autoregressive video diffusion models without sacrificing real-time inference speed or exceeding memory constraints. Specifically, the challenges include: 1) Full KV-cache attention’s memory footprint and attention cost grow linearly with rollout length, leading to GPU VRAM exhaustion and degraded throughput; 2) Sliding-window inference discards historical KV caches, losing long-term memory and causing content drift upon revisits; 3) Existing memory-augmented architectures require dedicated training and architectural modifications, increasing complexity; 4) 3D scene reconstruction methods introduce inference latency incompatible with real-time applications. Efficiently managing the KV cache to selectively retrieve relevant historical context and compress redundant information is essential to balancing memory fidelity and computational efficiency.

This work introduces several key innovations: 1) World Retrieval: a training-free mechanism that stores evicted KV-cache chunks indexed by camera pose and action state, and selectively retrieves the most relevant chunks based on similarity metrics, reinserting them into the attention window without re-encoding. This leverages the model’s intrinsic memory without architectural changes. 2) World Compression: a novel pruning strategy that uses key-key cosine similarity within each 3-frame chunk to identify and remove redundant tokens relative to an anchor frame, roughly halving per-chunk storage and enabling twice the historical coverage under fixed memory budgets. 3) A modular framework supporting multiple retrieval strategies (camera/action-based and query-based), demonstrating generality. 4) Empirical validation on two distinct world models with different training regimes, showing that WorldKV matches or surpasses full KV attention and memory-trained baselines in revisit fidelity while maintaining real-time throughput.

• �� World Retrieval: During sliding-window inference, KV-cache chunks evicted from the active attention window are stored in GPU/CPU memory, each indexed by the camera pose or cumulative action state at generation time. At generation, given the current camera/action state, a similarity function combining normalized squared L2 translation distance and geodesic rotation distance ranks stored chunks. The top-k most relevant chunks are retrieved and inserted into the attention window, preserving native attention computation without re-encoding.

• �� World Compression: For each 3-frame chunk, the first frame is designated as the anchor. Key vectors from non-anchor frames are compared via cosine similarity to anchor keys. Tokens with high similarity are pruned as redundant, while low-similarity tokens—representing newly revealed or dynamic regions—are retained. Compression is applied independently per transformer layer, resulting in compressed chunks approximately half the original size, enabling storage of roughly twice the number of chunks under fixed memory.

• �� The framework is agnostic to retrieval strategy; camera/action-based retrieval is used in experiments, but query-based importance scoring is also supported.

• �� Experiments involve evaluation on Matrix-Game-2.0 and LingBot-World-Fast, using metrics including LPIPS, PSNR, SSIM, FID, and throughput (FPS). Ablations explore compression ratios and retrieval budgets.

Experiments are conducted on two autoregressive video world models: LingBot-World-Fast (14B parameters, distilled from long-video teacher, full KV attention native) and Matrix-Game-2.0 (1.3B parameters, trained on short sequences, sliding-window native). The benchmark consists of 60 scene-trajectory pairs spanning diverse domains (indoor, outdoor, urban, natural), with manually designed long-horizon trajectories featuring repetitive revisits and loop closures for direct revisit consistency evaluation. Baselines include sliding-window and full KV attention, as well as memory-trained models WorldPlay and Yume-1.5. Metrics include LPIPS, PSNR, SSIM, FID for visual fidelity, and FPS for throughput. Ablation studies vary intra-chunk compression ratios and inter-chunk retrieval budgets to assess trade-offs between memory savings and revisit fidelity.

WorldKV achieves approximately 4.78 FPS on LingBot-World-Fast, close to sliding-window’s 5.05 FPS and significantly higher than full KV attention’s 2.36 FPS, while matching or surpassing full KV in LPIPS, PSNR, and FID. On Matrix-Game-2.0, WorldKV outperforms both sliding-window and full KV attention across all metrics, avoiding the compounded errors seen in full KV due to short-sequence training. Ablations show that retaining 25% of non-anchor tokens balances compression and fidelity, and increasing retrieval budget to cover more chunks improves revisit consistency. These results demonstrate that WorldKV effectively balances memory fidelity and computational efficiency without additional training.

WorldKV is applicable to real-time interactive video generation scenarios such as gaming, where consistent world states enhance player immersion; robotic simulation, enabling agents to explore and interact with stable virtual environments; and virtual reality systems requiring persistent scene memory for user navigation. Its training-free, modular design lowers deployment barriers and supports diverse domains and modalities. By improving long-term memory fidelity and inference speed, WorldKV facilitates more responsive and realistic interactive experiences, advancing applications in digital twins, intelligent agents, and immersive media.

WorldKV’s performance is inherently limited by the quality of the underlying pretrained world model, and does not address error accumulation over very long rollouts, which can degrade visual fidelity. CPU offloading reduces GPU memory usage but introduces host-device transfer latency that currently impedes real-time multi-minute inference. Additionally, camera/action-based retrieval may be less effective in scenarios with ambiguous or imprecise mappings between actions and viewpoints, potentially reducing memory recall accuracy. Addressing these limitations requires integration with training-based stabilization methods, hardware pipeline optimizations, and more robust retrieval strategies.