Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attention

Multimodal content generation has evolved from isolated audio or video synthesis to integrated models that generate synchronized audio-visual content. Early works like FoleyCrafter and MMaudio focused on aligning generated sounds with visual cues. Recent advances such as Ovi, Harmony, and LTX-2.3 utilize bidirectional cross-attention mechanisms to jointly generate coherent audio and video streams, significantly improving synchronization and realism. However, the complexity of attention mechanisms over long sequences remains a bottleneck, especially in real-time scenarios. Techniques like sparse attention and feature caching have been proposed to address intra-modal redundancy, but they often neglect the importance of cross-modal dependencies critical for maintaining synchronization. This gap motivates the need for methods that explicitly consider inter-modal interactions when optimizing computational efficiency.

Despite progress, current acceleration methods mainly target intra-modal redundancy, risking the loss of crucial cross-modal signals necessary for synchronization. The core challenge is to identify which attention computations are essential for preserving audio-video alignment and which are redundant. Traditional sparsification approaches may inadvertently remove important cross-modal interactions, leading to desynchronization or quality degradation. Moreover, existing cache strategies often ignore the dynamic nature of cross-modal dependencies, risking unsafe reuse and potential misalignment. Therefore, a key problem is designing an acceleration framework that dynamically protects synchronization-critical computations based on internal model signals, without requiring retraining or extensive profiling.

This work introduces a novel framework that exploits the inherent structure of bidirectional cross-attention to derive importance signals for acceleration. Key innovations include: 1) generating cross-modal saliency maps from attention responses to identify synchronization-critical tokens; 2) applying a protection mechanism that preserves dense attention for these tokens while sparsifying others; 3) employing a lagged saliency cache that leverages the stability of cross-modal interactions across steps to reduce computational overhead; 4) integrating feature change and saliency drift metrics to prevent unsafe cache reuse. These components collectively enable a training-free, adaptive acceleration strategy that maintains high fidelity and synchronization, surpassing prior intra-modal redundancy-based methods.

• �� Analyze bidirectional cross-attention to generate modality-specific saliency maps, highlighting tokens crucial for synchronization.
• �� Use these maps to select a subset of tokens for dense attention protection, ensuring critical cross-modal signals are preserved.
• �� Implement a lagged cache that stores saliency information from previous steps, exploiting the temporal stability of high-responsiveness regions.
• �� Combine feature change metrics and saliency drift measures to dynamically decide when to reuse cached features, avoiding unsafe reuse when dependencies shift.
• �� Apply a protected sparse attention mechanism, restoring full attention for protected tokens and sparse attention for others.
• �� Integrate these components into the inference pipeline, enabling efficient, synchronization-aware acceleration without retraining.

Experiments conducted on open-source models Ovi and LTX-2.3 involve generating synchronized videos with standard metrics: PSNR, SSIM, LPIPS, and audio-video synchronization scores (Sync-C, Sync-D). Baselines include dense models, sparse attention, and cache-based methods. The evaluation assesses quality preservation, speedup, and synchronization robustness. Hyperparameters such as protection ratio and cache update frequency are tuned for optimal trade-offs. Ablation studies analyze the impact of saliency-guided protection and cache reuse, confirming their effectiveness. Results show near 2× acceleration with minimal quality loss, outperforming existing acceleration methods in both fidelity and synchronization metrics.

The proposed method achieves 1.99× speedup on Ovi with PSNR rising from 21.76 to 25.69, SSIM above 0.83, and synchronization scores surpassing baseline methods. Similar improvements are observed on LTX-2.3, with PSNR from 18.76 to 20.87 and stable synchronization metrics. Ablation results confirm that saliency-guided protection reduces feature errors in critical regions, maintaining high fidelity. The stability of high-response regions across steps enables safe cache reuse, significantly reducing redundant computation. Overall, the approach effectively balances acceleration with the preservation of audio-video synchronization and quality.

This framework can be integrated into existing multimodal diffusion models for real-time applications such as virtual avatars, interactive entertainment, and remote conferencing. Its training-free nature allows easy deployment, requiring only attention analysis and saliency-guided protection modules. The method enhances efficiency without sacrificing synchronization, making it suitable for industry-scale content creation, live streaming, and virtual reality scenarios. Long-term, it paves the way for highly efficient, high-fidelity multimodal synthesis systems capable of operating under strict latency constraints, fostering innovations in immersive media and AI-driven content production.

The reliance on stable cross-modal attention patterns may limit robustness in highly dynamic or noisy environments. The method might miss subtle but important signals not reflected in high attention responses, risking minor desynchronization. Computational overhead from saliency estimation and cache management, though limited, could pose challenges for extremely large models or ultra-low latency applications. Future work should focus on adaptive importance learning, multi-modal extension, and hardware optimization to address these limitations.