JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
JavisDiT++ achieves high-quality audio-video generation using MS-MoE and TA-RoPE, surpassing existing methods.
Key Findings
Methodology
JavisDiT++ employs a modular MS-MoE design to enhance single-modality generation quality and a TA-RoPE strategy for frame-level synchronization of audio and video tokens. Additionally, an AV-DPO method aligns model outputs with human preferences across quality, consistency, and synchrony dimensions. Built on Wan2.1-1.3B-T2V, the model achieves state-of-the-art performance with only about 1M public training entries.
Key Results
- In JavisBench, JavisDiT++ significantly outperforms UniVerse-1 in audio-video quality and consistency, reducing FVD to 141.5 and FAD to 5.5.
- The TA-RoPE strategy significantly improves audio-video synchronization metrics, reducing DeSync to 0.832.
- Ablation studies validate the effectiveness of MS-MoE and TA-RoPE modules, demonstrating their contributions across different dimensions.
Significance
This research sets a new milestone in the field of joint audio-video generation, significantly improving generation quality and synchronization. By introducing human preference alignment, the model generates audio-video content that better aligns with human aesthetics and needs, offering new possibilities for multimodal generation technology in fields like film, gaming, and virtual reality.
Technical Contribution
JavisDiT++ innovates in model architecture with MS-MoE and TA-RoPE strategies, providing a more concise and efficient solution compared to existing complex dual-stream models. Additionally, it is the first to introduce human preference alignment in audio-video generation, significantly enhancing the quality and consistency of generated content.
Novelty
JavisDiT++ is the first to apply human preference alignment to joint audio-video generation and achieves precise temporal synchronization with TA-RoPE, which is more direct and effective compared to traditional ST-Prior and cross-attention mechanisms.
Limitations
- The model may experience performance degradation in long-duration audio-video generation tasks, requiring further optimization.
- The current preference dataset may not be comprehensive enough, affecting the model's generalizability.
Future Work
Future work could expand to larger datasets, explore more complex multimodal interaction mechanisms, and further optimize human preference alignment strategies to enhance the diversity and adaptability of generated content.
AI Executive Summary
Joint audio-video generation is a crucial task in the field of multimodal generation, with existing open-source methods still lacking in generation quality and synchronization. JavisDiT++ introduces MS-MoE design and TA-RoPE strategy, significantly enhancing the quality and synchronization of audio-video generation. Additionally, the AV-DPO method aligns model outputs with human preferences, making the generated content more in line with human aesthetics and needs. Experimental results show that JavisDiT++ surpasses existing methods in multiple metrics, particularly in audio-video synchronization and consistency. However, the model still requires further optimization for long-duration generation tasks, and future work will continue to explore more complex multimodal interaction mechanisms.
Deep Analysis
Background
In recent years, AI-generated content (AIGC) has rapidly expanded from text-to-image generation to video and audio domains. Joint audio-video generation (JAVG) aims to produce synchronized and semantically aligned sound and vision from textual descriptions. Despite progress, existing open-source methods still fall short in generation quality and synchronization.
Core Problem
Existing audio-video generation methods struggle with generation quality, temporal synchronization, and alignment with human preferences. The challenge lies in producing high-quality, synchronized audio-video content that aligns with human aesthetics.
Innovation
JavisDiT++ introduces MS-MoE design to enhance single-modality generation quality and a TA-RoPE strategy for frame-level synchronization. Additionally, the AV-DPO method aligns model outputs with human preferences, covering quality, consistency, and synchrony dimensions.
Methodology
- �� MS-MoE Design: Enables cross-modal information exchange through multi-head self-attention layers, aggregating intra-modal information via separate FFN layers. • TA-RoPE Strategy: Aligns position IDs of audio and video tokens on a unified temporal axis for frame-level synchronization. • AV-DPO Method: Evaluates generated audio-video samples using various reward models and adopts normalized modality-aware ranking to select winning-losing pairs.
Experiments
Experiments were conducted on JavisBench and JavisBench-mini, covering 11 metrics across various audio-video dimensions, including quality, consistency, and synchrony. The model significantly outperforms existing methods using only about 1M public training entries.
Results
JavisDiT++ significantly outperforms UniVerse-1 in audio-video quality and consistency, reducing FVD to 141.5 and FAD to 5.5. The TA-RoPE strategy significantly improves audio-video synchronization metrics, reducing DeSync to 0.832.
Applications
JavisDiT++ has broad application potential in fields like film production, game development, and virtual reality, capable of generating high-quality, synchronized audio-video content that aligns with human preferences.
Limitations & Outlook
The model may experience performance degradation in long-duration audio-video generation tasks, requiring further optimization. Additionally, the current preference dataset may not be comprehensive enough, affecting the model's generalizability.
Plain Language Accessible to non-experts
Imagine you're in a kitchen, and JavisDiT++ is like a smart chef assistant. It adjusts the ingredients and cooking times based on your taste preferences to ensure every dish is perfect. The MS-MoE design is like different chefs, each specializing in a different cuisine, while the TA-RoPE strategy ensures each dish is served at the right time. The AV-DPO method acts like a tasting panel, ensuring each dish meets your taste.
ELI14 Explained like you're 14
Imagine you're playing a super cool game, and JavisDiT++ is like the smart assistant in the game. It helps you generate super realistic sound effects and visuals, making you feel like you're really there. The MS-MoE is like different characters in the game, each with different skills, while the TA-RoPE is like magic that makes these characters work perfectly together. The AV-DPO is like a super skilled referee, ensuring the game experience matches your preferences.
Glossary
MS-MoE (Modality-Specific Mixture-of-Experts)
A modular design that enhances single-modality generation quality.
Used to improve the quality of audio-video generation.
TA-RoPE (Temporal-Aligned Rotary Position Encoding)
A strategy for achieving frame-level synchronization of audio and video tokens.
Ensures temporal synchronization of audio-video content.
AV-DPO (Audio-Video Direct Preference Optimization)
A method for aligning model outputs with human preferences.
Enhances the quality and consistency of generated content.
FVD (Fréchet Video Distance)
Measures the distance between generated and real videos.
Used to evaluate the quality of generated videos.
FAD (Fréchet Audio Distance)
Measures the distance between generated and real audio.
Used to evaluate the quality of generated audio.
Open Questions Unanswered questions from this research
- 1 How to maintain high quality and synchronization in long-duration audio-video generation?
- 2 How to expand the preference dataset to improve model generalizability?
Applications
Immediate Applications
Film Production
Can be used to generate high-quality audio-video content, enhancing the efficiency and effect of film production.
Long-term Vision
Virtual Reality
Applied in virtual reality to provide more immersive experiences, requiring solutions for real-time generation challenges.
Abstract
AIGC has rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio-video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and dataset are released at https://JavisVerse.github.io/JavisDiT2-page.