ShadowDancer: Teaching Video World Models Any Action by Learning Unified Dynamics Representations from a Video and Its Shadow
ShadowDancer learns unified dynamic representations from video-shadow pairs, enabling precise, label-free action transfer across diverse scenarios.
Key Findings
Methodology
ShadowDancer employs shadow pairs—video pairs that replay the same dynamics with different appearances—constructed via a large-scale Shadow Library. Using cross-shadow prediction, a latent action model (LAM) is trained to encode shared dynamics by predicting one shadow from the other, enforcing invariance. The approach integrates a pretrained video diffusion backbone, converted into a block-causal autoregressive generator, enabling long-horizon, variable-length action rollouts. This framework eliminates the need for action labels, motion estimators, or fine-tuning, facilitating direct action transfer and reuse across environments.
Key Results
- Across multiple dynamics families—including human motion, robotic manipulation, and open-world gameplay—ShadowDancer achieves an average blinded rollout win rate of 86%, outperforming state-of-the-art latent-action models like Olaf-World. Quantitative metrics show PSNR improvements from 18.2 to 22.4 and LPIPS reductions from 0.288 to 0.184, indicating higher fidelity and perceptual quality. In cross-environment action transfer, the model faithfully reproduces demonstrated dynamics with minimal artifacts, demonstrating strong generalization.
- In long-term action rollouts, the model maintains high consistency, with PSNR and LPIPS metrics indicating stable performance over extended sequences. The separation of dynamics from appearance allows robust transfer in diverse scenarios, including unseen environments and different scene configurations. The results validate the effectiveness of shadow pairs and the cross-shadow prediction training paradigm in learning invariant, reusable action representations.
- The approach significantly reduces reliance on explicit labels or motion estimation, simplifying the pipeline and broadening applicability. It demonstrates that a unified, label-free dynamic representation can be learned from complex, multi-source video data, paving the way for more flexible and scalable interactive video world models.
Significance
This work addresses a fundamental challenge in video-based control: how to learn dynamic representations that generalize across appearances and environments without supervision. By introducing shadow pairs and a self-supervised training scheme, it overcomes the entanglement of appearance and motion that hampers traditional latent models. The resulting unified dynamics representation enables precise, label-free control, which is crucial for advancing interactive virtual worlds, robotics, and simulation. Its ability to transfer actions seamlessly across diverse scenarios marks a significant step toward autonomous, adaptable systems that learn from raw video data, reducing reliance on costly annotations and handcrafted signals. This innovation opens new avenues for scalable, general-purpose video understanding and control, with broad implications for AI research and industry applications.
Technical Contribution
The core technical contribution lies in the shadow pairing protocol and cross-shadow prediction mechanism. Shadow pairs are constructed by rendering the same dynamics with independently resampled appearances, ensuring the shared dynamics are the only invariant factor. The cross-shadow prediction enforces that the latent action encodes this invariant dynamics by predicting one shadow from the other, grounded in a variational framework. This guarantees the learned representation is disentangled from appearance. The integration with a pretrained video diffusion backbone, transformed into a block-causal generator, enables long-horizon, variable-length action generation. The architecture supports end-to-end training without labels, leveraging large-scale multi-source data, and provides formal guarantees on the identifiability of shared dynamics. This approach fundamentally differs from prior latent-action models by embedding invariance into the data rather than regularizing the latent space post hoc.
Novelty
This is the first work to leverage shadow pairs for self-supervised, label-free learning of dynamic representations that are invariant to appearance. Unlike previous methods relying on explicit motion signals or symbolic commands, ShadowDancer constructs a data-driven pairing protocol that guarantees invariance by design. Its cross-shadow prediction training enforces the disentanglement of dynamics from appearance without supervision. The combination of large-scale shadow library construction, invariant representation learning, and integration with a pretrained diffusion model for interactive control constitutes a novel paradigm in video understanding and generation. This approach fundamentally shifts how dynamic actions are learned and transferred, enabling broad generalization across diverse scenarios.
Limitations
- The construction of shadow pairs requires extensive multi-source data and computational resources, limiting immediate deployment in resource-constrained settings.
- The method's performance may degrade in highly complex or rapidly changing environments where shadow pair generation becomes challenging.
- Current focus is on static scenes and a limited set of dynamics; extending to highly dynamic, unstructured, or real-world videos remains an open challenge.
Future Work
Future directions include improving the efficiency of shadow pair generation, reducing data requirements, and extending the framework to handle more complex, real-world scenarios. Incorporating multi-modal signals such as audio or haptic feedback could enrich dynamic representations. Additionally, integrating reinforcement learning to optimize action quality and exploring multi-action composition will further enhance the system's versatility. Scaling to real-time applications and deploying in robotics or interactive entertainment are promising long-term goals.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
Imagine you’re watching a dance performance. The dancer’s movements are the same, but the costumes and lighting change each time. ShadowDancer is like filming the same dance twice under different costumes and lighting, then analyzing what makes the dance unique. By comparing these two videos, the system learns the core dance moves that stay the same regardless of appearance. Later, it can perform or imitate the dance in new costumes or settings, just by understanding those core moves. This is similar to how you might learn a dance step once and then do it perfectly no matter what you’re wearing or where you are. The system’s ability to focus on the dance itself, ignoring the changing costumes, makes it very flexible and powerful for teaching virtual characters or robots new actions that look natural in any environment.
Abstract
We present ShadowDancer, a novel approach to any-action, frame-level control of interactive video world models. The obstacle is representational: existing interfaces either encode an action loosely, leaving how it unfolds for the model to improvise, or encode it exactly through structured signals that serve one family and are hard to acquire, so precise control across diverse dynamics remains impractical. Demonstration videos are the natural remedy, specifying any dynamics frame by frame; yet a video shows its dynamics only through one particular appearance, a single shadow of the underlying dynamics, so actions learned from demonstrations transfer poorly to new scenes. ShadowDancer addresses this with two key innovations: (1) shadow pairs, video pairs that replay the same dynamics under independently resampled appearance, constructed at scale by our Shadow Library, so that a dynamics family becomes controllable exactly when such pairs can be constructed for it; and (2) cross-shadow prediction, which learns actions by predicting one shadow from the other, so that whatever the pairing resamples is discarded by construction and whatever it preserves becomes the action, yielding a unified dynamics representation that drives a block-causal world model. Any demonstrated clip thus becomes a reusable action asset, replayed in new environments without action labels, motion estimators, or fine-tuning. Experiments demonstrate improved action transfer and long action rollout over strong latent-action and interactive world model baselines across diverse dynamics families, with an average blinded win rate of 86% in rollout comparisons. We show video results at https://ShadowDancer-1.github.io