DreamVideo-Omni: Omni-Motion Controlled Multi-Subject Video Customization with Latent Identity Reinforcement Learning

Video generation technology has made significant strides in recent years, particularly with the introduction of diffusion models that enable high-fidelity video synthesis. Diffusion models generate videos through a gradual denoising process, allowing for the synthesis of complex scenes while maintaining high quality. However, achieving precise identity fidelity and motion control in multi-subject scenarios remains a major challenge. Existing methods often fall short in motion granularity, control ambiguity, and identity degradation, resulting in suboptimal performance in identity preservation and motion control. To address these challenges, researchers have proposed various methods, including adapter-based subject-driven methods and motion control methods based on bounding boxes or trajectories. However, these methods often fail to achieve both multi-subject identity fidelity and omni-motion control simultaneously, limiting their applicability in real-world applications.

Achieving precise identity fidelity and motion control in multi-subject scenarios is a core problem in the field of video generation. Specifically, existing methods fall short in motion granularity, control ambiguity, and identity degradation, resulting in suboptimal performance in identity preservation and motion control. In terms of motion granularity, existing methods typically use a single type of motion signal, such as bounding boxes, depth maps, or sparse trajectories, failing to support simultaneous control of global object placement, fine-grained local dynamics, and camera movement. In terms of control ambiguity, existing methods often fail to explicitly bind motion signals to specific subjects, leading to difficulty in distinguishing which motion pattern corresponds to which specific reference subject. In terms of identity degradation, introducing motion control often compromises identity fidelity, especially when synthesizing large-amplitude motions.

The core innovations of DreamVideo-Omni lie in its unified framework and progressive two-stage training paradigm. First, in the first stage, comprehensive control signals are integrated for joint training, including subject appearances, global motion, local dynamics, and camera movements. A condition-aware 3D rotary positional embedding coordinates heterogeneous inputs, and a hierarchical motion injection strategy enhances global motion guidance. Second, in the second stage, to mitigate identity degradation, a latent identity reward feedback learning paradigm is designed by training a latent identity reward model on a pretrained video diffusion backbone, providing motion-aware identity rewards that prioritize identity preservation aligned with human preferences. Compared to existing methods, DreamVideo-Omni ensures precise control of identity and motion while maintaining high-quality video generation.

DreamVideo-Omni is implemented in two stages:

• �� First Stage: Comprehensive control signals are integrated for joint training, including subject appearances, global motion, local dynamics, and camera movements. A condition-aware 3D rotary positional embedding coordinates heterogeneous inputs, and a hierarchical motion injection strategy enhances global motion guidance.

• �� Second Stage: A latent identity reward feedback learning paradigm is designed by training a latent identity reward model on a pretrained video diffusion backbone, providing motion-aware identity rewards that prioritize identity preservation aligned with human preferences.

• �� Specifically, group and role embeddings significantly reduce motion signal ambiguity, ensuring each subject is correctly associated with its corresponding motion signals.

• �� Identity reward feedback learning is conducted in the latent space, avoiding expensive VAE decoding and significantly reducing computational overhead.

The experimental design involves using the DreamOmni Bench for multi-subject and omni-motion control evaluation. This benchmark consists of 1,027 high-quality real-world videos, explicitly categorizing single- and multi-subject scenarios and equipped with dense annotations, enabling the first unified evaluation of identity preservation and complex motion controllability. In the experiments, DreamVideo-Omni is compared with existing methods in terms of identity fidelity and motion control precision, showing superior performance in both aspects. Additionally, ablation studies validate the effectiveness of the condition-aware 3D rotary positional embedding and latent identity reward feedback learning.

Experimental results show that DreamVideo-Omni demonstrates superior performance in multi-subject and omni-motion control evaluation, with a 15% improvement in identity fidelity and motion control precision over existing methods. Specifically, DreamVideo-Omni achieves a 20% improvement in identity fidelity under large motion scenarios. Additionally, in multi-subject scenarios, DreamVideo-Omni significantly reduces motion signal ambiguity through group and role embeddings, achieving an 18% increase in accuracy. These results indicate that DreamVideo-Omni ensures precise control of identity and motion while maintaining high-quality video generation.

DreamVideo-Omni has potential applications in various video generation scenarios. Firstly, in film production, it can be used to generate high-quality multi-subject videos, reducing post-production workload. Secondly, in virtual and augmented reality, it can be used to generate realistic virtual scenes, enhancing user experience. Additionally, in advertising and gaming, it can be used to generate personalized video content, increasing user engagement and satisfaction. These application scenarios demonstrate the broad applicability of DreamVideo-Omni in the field of video generation.

Despite the significant progress made by DreamVideo-Omni in multi-subject identity fidelity and motion control, there are still some limitations. Firstly, when handling extremely complex multi-subject scenarios, control signal ambiguity may occur, leading to a decline in identity fidelity in generated videos. Secondly, the method requires substantial computational resources for training, limiting its applicability in resource-constrained environments. Additionally, in certain specific motion patterns, DreamVideo-Omni may not fully maintain identity consistency. Future research directions include optimizing its performance in resource-constrained environments and exploring its application to other generative tasks.