Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation
Ovi employs twin DiT backbones with blockwise bidirectional cross-attention and scaled-RoPE to unify audio-video generation, trained on hundreds of thousands of hours of raw data.
Key Findings
Methodology
Ovi utilizes two symmetric DiT (diffusion transformer) backbones for audio and video, integrated with blockwise bidirectional cross-attention layers. Each transformer block exchanges temporal (via scaled-RoPE embeddings) and semantic information (via bidirectional cross-attention). The training involves two stages: first, pretraining the audio backbone on large-scale, richly captioned audio datasets from scratch to learn realistic sound effects and speaker traits; second, joint fine-tuning of both backbones on paired audio-video data to learn synchronization without sacrificing unimodal fidelity. A single frozen T5 encoder conditions both modalities, simplifying semantic control. The core innovations include block-level cross-attention and scaled-RoPE, which enable precise temporal alignment and natural synchronization, eliminating the need for post-processing or heuristic alignment.
Key Results
- On Verse-Bench, Ovi outperforms JavisDiT and UniVerse-1 in human preference tests, with over 75% favorability across audio quality, video quality, and synchronization. Quantitative metrics such as FDPANNs (18.03), FDVGG, and CLAP confirm high content realism and semantic alignment. Ablation studies demonstrate that blockwise cross-attention and RoPE scaling are critical for achieving accurate synchronization. The model generates 5-second, 720×720 videos at 24 fps, with realistic speech and sound effects, matching professional-grade content quality.
- Training on large-scale datasets, Ovi produces coherent, high-fidelity audio and video, with natural lip-sync and scene matching. The model’s architecture ensures efficient end-to-end generation, with results comparable to specialized models but with the added benefit of unified multimodal control. Human preference surveys validate the perceptual superiority of Ovi, indicating its potential as a foundational model for diverse applications.
- The multi-stage training process—initial audio pretraining followed by joint fine-tuning—enables the model to balance content richness and synchronization accuracy. The experimental results highlight the importance of block-level fusion and temporal scaling, setting new benchmarks in open multimodal generation research. Overall, Ovi advances the state-of-the-art, demonstrating scalable, high-quality, synchronized audio-video synthesis.
Significance
This work addresses fundamental challenges in unified multimodal content creation, achieving natural, synchronized audio-video generation without complex post-processing or task-specific pipelines. It significantly reduces the complexity of multi-stage workflows, paving the way for real-time virtual content production, immersive entertainment, and interactive applications. The model’s architecture and training strategy provide a scalable framework adaptable to various content types and lengths, fostering innovation in virtual reality, film, gaming, and beyond. By enabling high-fidelity, synchronized multimodal outputs, Ovi bridges the gap between research and practical deployment, promising a new era of intelligent content synthesis.
Technical Contribution
The study introduces a symmetric twin backbone architecture with blockwise bidirectional cross-attention, integrated with scaled-RoPE positional embeddings for precise temporal alignment. This design ensures high-fidelity synchronization and content consistency. The training pipeline combines large-scale self-supervised pretraining on diverse audio data with targeted joint fine-tuning on paired audio-video datasets, optimizing both unimodal quality and multimodal coherence. The use of a single frozen T5 encoder for semantic conditioning simplifies the architecture and enhances control. These innovations collectively enable end-to-end, high-quality, synchronized multimodal generation, surpassing existing methods that rely on heuristics or separate pipelines.
Novelty
Ovi is the first large-scale, end-to-end audio-video generator employing blockwise bidirectional cross-attention and scaled-RoPE embeddings for precise temporal synchronization. Unlike prior models that depend on post-processing, pretraining, or architecture mismatches, Ovi’s symmetric backbone design and integrated fusion mechanism directly address the core challenge of natural multimodal synchronization. Its training strategy and architecture set new standards for open multimodal content generation, filling a significant gap in the field.
Limitations
- Despite high synchronization accuracy, the model may struggle in highly dynamic or fast-moving scenes, where temporal misalignments could occur. The reliance on large paired datasets increases training costs and limits deployment in low-resource settings. Additionally, the current architecture is optimized for short clips (~5 seconds), and extending to longer videos remains a challenge due to computational constraints. Future work should focus on improving robustness, reducing data dependency, and scaling to longer sequences.
Future Work
Future directions include enhancing model robustness in complex scenes, reducing computational costs for longer video generation, and integrating more diverse modalities such as text and 3D data. Exploring unsupervised or semi-supervised training methods could lower data requirements. Additionally, incorporating user controls and real-time capabilities will expand practical applications in entertainment, education, and virtual environments. Continued research aims to refine synchronization accuracy, content diversity, and scalability, pushing the boundaries of unified multimodal synthesis.
AI Executive Summary
The rapid growth of virtual content demands seamless integration of audio and visual modalities, yet existing solutions often rely on multi-stage pipelines or post-processing adjustments, limiting real-time applicability and content fidelity. Addressing this challenge, the Ovi model introduces a novel framework that unifies audio-video generation in a single end-to-end process. Central to its design are two symmetric diffusion transformer backbones, each dedicated to one modality, interconnected through blockwise bidirectional cross-attention layers. This architecture enables real-time, precise synchronization without heuristic post-processing, a significant leap over prior methods.
The core innovation lies in the use of scaled-RoPE embeddings, which align the temporal resolutions of audio and video streams, ensuring their natural coupling. During training, the model undergoes a two-stage process: first, pretraining the audio backbone on hundreds of thousands of hours of richly captioned audio data to master speech and sound effects; second, joint fine-tuning on paired audio-video datasets to learn synchronization. This approach leverages large-scale self-supervised learning and multimodal fusion, resulting in high-quality, cinematic-grade content.
Experimental results on the Verse-Bench dataset demonstrate that Ovi surpasses existing open-source models like JavisDiT and UniVerse-1, with preference rates exceeding 75% in human surveys. The generated videos exhibit realistic speech, synchronized sound effects, and scene-appropriate visuals at 720×720 resolution and 24 fps, validating the model’s effectiveness. The findings suggest that this unified approach not only simplifies the content creation pipeline but also sets new standards for multimodal synthesis, with broad implications for entertainment, virtual reality, and interactive media. Despite current limitations, such as computational costs and scene complexity, the framework offers a scalable, flexible foundation for future research and practical deployment in diverse multimedia applications.
Deep Analysis
Background
多模态内容生成技术在过去十年快速发展，代表性工作包括OpenAI的Sora、Wan等基于潜在空间的扩散模型，以及专注于文本到视频（T2V）、音频到视频（A2V）和视频到音频（V2A）任务的模型。这些方法多采用预训练+微调策略，或依赖后处理实现模态同步，但存在同步精度不足、模型复杂、难以扩展的问题。Google的Veo3表现优异，但为闭源系统，限制了研究的透明度和复用性。现有开源模型多在单模态或简单多模态任务中表现有限，难以满足真实场景中的内容一致性和同步需求。
Core Problem
核心挑战在于如何在单一模型中实现音频与视频的自然同步，避免多阶段、多管线的复杂流程。现有方法多依赖后处理或特定任务优化，难以在多场景、多内容类型下保持高质量和一致性。尤其是在长时视频、多样音效和丰富语者信息的生成中，模态间的时间对齐和语义一致性成为瓶颈。这限制了多模态内容的真实感和沉浸感，亟需一种端到端、可扩展的解决方案。
Innovation
本文提出的Ovi模型具有多项创新：1）引入块级双向交叉注意机制，实现模态间的实时信息交换；2）采用scaled-RoPE嵌入解决时间对齐问题，确保音视频在时间维度上的一致性；3）对称的骨架设计，避免参数膨胀和后处理依赖；4）多阶段训练策略，结合大规模预训练和微调，提升模型泛化能力。这些创新共同推动了多模态同步生成的技术边界，解决了以往模型在同步精度和内容丰富性上的不足。
Methodology
- �� 构建两个对称的扩散变换器骨架，分别处理音频和视频模态。• 在每个Transformer块中引入双向交叉注意机制，使两个模态可以互相学习和同步。• 利用scaled-RoPE嵌入对时间位置进行缩放，确保音视频在时间维度上的对齐。• 采用单一冻结的T5编码器，将语义条件统一调控，简化训练流程。• 训练分两个阶段：先用大规模带字幕的音频数据训练音频骨架，再在配对的音视频数据上微调融合层。• 通过块级交换时间和语义信息，实现高效同步，无需后处理或多模预训练。• 最终模型能生成长达5秒、720×720分辨率的视频，音频中既有语音也有丰富音效。
Experiments
采用Verse-Bench数据集进行评估，比较模型在音频、视频质量和同步性上的表现。使用偏好率调查，结果显示Ovi优于JavisDiT和UniVerse-1，偏好率超过75%。指标包括FDPANNs（18.03）、FDVGG、Inception Score和CLAP，验证内容的自然性和语义一致性。 Ablation研究确认块级交叉注意和RoPE缩放是同步的关键。模型在多项指标上与专用模型相当，验证其作为多模态生成基础模型的潜力。
Results
在Verse-Bench上，Ovi在音频质量（FDPANNs=18.03）、视频质量和同步性方面均优于对比模型，偏好率达75%以上。生成的内容在保持高分辨率和丰富细节的同时，实现了自然流畅的同步。模型还在多个指标上与专用模型持平，验证了其多模态融合能力。 Ablation实验显示，块级交叉注意和RoPE缩放是实现高精度同步的核心技术。整体结果表明，该模型在多模态内容生成领域具有突破性潜力。
Applications
该模型适用于虚拟现实、影视制作、交互娱乐等行业，支持高质量、同步的多模态内容生成。只需提供文本提示，模型即可自动生成符合场景需求的音视频内容。未来可结合实时交互系统，实现虚拟主播、虚拟演唱会等应用，极大丰富虚拟内容生态。
Limitations & Outlook
模型在极端运动场景下的同步鲁棒性仍需提升，长视频生成时计算成本较高，数据依赖大规模配对样本，限制了其普适性。未来需优化模型结构，降低计算成本，增强多场景适应能力。
Plain Language Accessible to non-experts
想象你在一家大型工厂里，工厂里有两个机器人，一个负责制造声音，另一个负责制作画面。它们需要合作，确保声音和画面同步，就像一场精彩的演出。以前，工厂里的机器人各自工作，之后还要花时间调整，确保同步。而这次，科学家设计了一个特殊的系统，让两个机器人可以实时交流信息，彼此协调。这个系统就像两个机器人之间有一条隐形的线，能让它们同时知道对方在做什么。这样，工厂里的声音和画面就能像专业演员一样自然配合，节省了很多调整时间，也让内容看起来更真实、更精彩。这个新系统不仅提高了效率，还让虚拟世界变得更丰富、更逼真，就像一场完美的演出一样。
ELI14 Explained like you're 14
想象你在看一场动画电影，里面的角色会说话、跑动，背景音乐也在配合场景变化。以前，要让声音和画面完美同步，通常需要很多繁琐的步骤，比如先做完画面，再配上声音，或者用特殊软件调整时间。现在，科学家发明了一种新方法，就像两个聪明的朋友一样，他们可以一边说话一边跳舞，完全同步，不用后续调整。这种方法用一种叫“对称的模型”来让声音和画面同时学习，确保它们在时间上对齐。它还用一种特别的“缩放技巧”让两者的节奏一致。这样一来，生成的动画电影就更自然，像真人一样流畅。这个技术可以用在虚拟主播、游戏、电影制作中，让虚拟内容变得更逼真、更有趣。
Glossary
Diffusion Transformer（扩散变换器）
结合扩散模型与Transformer架构的生成模型，用于高质量内容合成。
论文中用以处理音视频的生成任务。
Blockwise Bidirectional Cross-Attention（块级双向交叉注意）
在Transformer中实现模态间信息交换的机制，支持同步。
确保音视频内容的实时协调。
scaled-RoPE（缩放的旋转位置编码）
一种时间位置编码技术，通过缩放频率解决不同模态时间对齐问题。
使音视频在时间维度上保持一致。
Symmetric Backbone（对称骨架）
结构上对称的两个Transformer，用于处理不同模态，保证参数一致性。
实现模态同步的关键设计。
End-to-End（端到端）
从输入到输出全部在单一模型中完成，无需中间步骤。
实现音视频同步的完整流程。
Open Questions Unanswered questions from this research
- 1 在高速运动或复杂场景中保持同步的鲁棒性仍需提升，模型在极端条件下表现尚不理想。
- 2 大规模配对数据的收集成本高，限制了模型的普适性和推广。
- 3 长时视频生成的效率和内容多样性仍待优化，未来需结合更高效的训练策略。
Applications
Immediate Applications
虚拟主播与虚拟演唱会
利用Ovi模型生成同步的虚拟人物音视频内容，支持实时互动，提升虚拟娱乐体验。
影视后期制作
自动生成逼真音效和同步画面，降低后期成本，加快制作流程。
Long-term Vision
虚拟现实与沉浸式体验
实现高度真实的虚拟场景，支持交互式内容生成，推动虚拟世界的普及与发展。
Abstract
Audio-video generation has often relied on complex multi-stage architectures or sequential synthesis of sound and visuals. We introduce Ovi, a unified paradigm for audio-video generation that models the two modalities as a single generative process. By using blockwise cross-modal fusion of twin-DiT modules, Ovi achieves natural synchronization and removes the need for separate pipelines or post hoc alignment. To facilitate fine-grained multimodal fusion modeling, we initialize an audio tower with an architecture identical to that of a strong pretrained video model. Trained from scratch on hundreds of thousands of hours of raw audio, the audio tower learns to generate realistic sound effects, as well as speech that conveys rich speaker identity and emotion. Fusion is obtained by jointly training the identical video and audio towers via blockwise exchange of timing (via scaled-RoPE embeddings) and semantics (through bidirectional cross-attention) on a vast video corpus. Our model enables cinematic storytelling with natural speech and accurate, context-matched sound effects, producing movie-grade video clips. All the demos, code and model weights are published at https://aaxwaz.github.io/Ovi