MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation
MUNI employs an end-to-end multimodal latent diffusion framework for arbitrary-to-arbitrary generation, surpassing baselines in both conditional and unconditional tasks.
Key Findings
Methodology
MUNI innovatively extends latent diffusion models to handle multimodal arbitrary-to-arbitrary generation within an end-to-end training paradigm. The architecture includes modality-specific encoders, expressive decoders, and a single shared flow-based prior, jointly optimized under one objective. Unlike traditional two-stage approaches that precompute a frozen latent space and then fit a prior, MUNI trains all components simultaneously, enabling dynamic learning of the latent space. To address the inadequacy of standard multimodal variational inference aggregation rules when coupled with learned priors and expressive decoders, the authors propose a routed training objective. This design ensures the shared latent space maintains coherence across modalities, predictive sufficiency of subset latents, and minimal content redundancy. The structural choices in the objective facilitate a minimal-sufficiency characterization, leading to improved multimodal coherence and generalization.
Key Results
- On PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark, MUNI outperforms state-of-the-art baselines in conditional generation, with improvements exceeding 15% in key metrics. In unconditional generation, it demonstrates superior cross-modal coherence, significantly surpassing competing models. Ablation studies confirm that end-to-end training and the routed objective substantially enhance modal consistency and latent space expressiveness.
- The model maintains stable performance across various modal combinations, showing strong generalization. Quantitative metrics such as FID scores, CLIP similarity, and cross-modal consistency indices validate its effectiveness.
Significance
This work advances the field of multimodal generative modeling by removing the dependency on paired text data and providing a unified framework for arbitrary-to-arbitrary generation. Its end-to-end training strategy and latent space structuring offer new theoretical insights and engineering possibilities, enabling more coherent and flexible multimodal content creation. The approach addresses longstanding challenges in multimodal fusion, content fidelity, and generalization, with broad implications for multimedia synthesis, virtual reality, and human-AI interaction.
Technical Contribution
Technically, MUNI introduces a comprehensive multimodal latent diffusion architecture, integrating modality-specific encoders, decoders, and a shared flow prior within a single training loop. The routed training objective enforces coherence, minimality, and sufficiency constraints on the latent space, backed by a theoretical minimal-sufficiency characterization. This design enhances the expressiveness and robustness of the latent space, enabling high-quality joint multimodal generation without reliance on paired data, and opens new avenues for scalable, flexible multimodal AI systems.
Novelty
This is the first work to extend latent diffusion models to fully end-to-end multimodal arbitrary-to-arbitrary generation, employing a routed training objective to optimize the latent space structure. Unlike prior models that depend on text alignment or deterministic mappings, MUNI achieves modality-agnostic, unpaired multimodal synthesis, representing a significant leap in the field. Its innovative latent space design and training strategy set it apart from existing approaches.
Limitations
- Training requires large-scale, diverse multimodal datasets, which can be costly and challenging to curate, especially for high-dimensional modalities.
- Computational complexity remains high, particularly for high-resolution images or long audio sequences, limiting real-time applications.
- The current model's performance may degrade in scenarios with highly imbalanced or noisy modality inputs, indicating a need for robustness improvements.
Future Work
Future research will focus on reducing data and computational requirements, exploring more efficient training algorithms, and enhancing robustness against noisy or incomplete modalities. Expanding the model's capacity to handle higher-dimensional data and more complex multimodal tasks, such as video synthesis, is also planned. Additionally, integrating this framework into practical applications like multimedia content creation, virtual assistants, and immersive environments will be key directions.
AI Executive Summary
Multimodal content generation remains a complex challenge in artificial intelligence, hindered by the reliance on paired datasets and modality-specific models. Traditional approaches often struggle to produce coherent, high-quality outputs across diverse modalities, limiting their practical utility. Addressing these issues, the paper introduces MUNI, a novel end-to-end multimodal latent diffusion framework designed for arbitrary-to-arbitrary generation. Unlike prior models that depend heavily on text-paired data or deterministic mappings, MUNI unifies modality-specific encoders, expressive decoders, and a single shared flow-based prior within a cohesive training scheme. The key innovation lies in the routed training objective, which enforces the latent space to satisfy multiple criteria—coherence across modalities, predictive sufficiency of sub-latents, and minimal content redundancy. This structure ensures the latent space is both expressive and efficient, enabling high-fidelity, cross-modal content synthesis.
Experimental results demonstrate MUNI's superiority over existing baselines on benchmarks like PolyMNIST-Quadrant-Labels and large-scale image-text-audio datasets. It achieves over 15% improvements in conditional generation metrics and significantly enhances unconditional cross-modal coherence. These findings highlight the model's ability to generate consistent, high-quality multimodal content without requiring paired data, marking a substantial step forward in the field. The framework's flexibility and robustness open new avenues for applications in multimedia creation, virtual reality, and human-AI interaction, promising more natural and versatile multimodal systems.
Despite its strengths, MUNI faces challenges such as high computational costs and reliance on large datasets, which may limit scalability. Future work aims to optimize training efficiency, improve robustness, and extend capabilities to more complex data types like videos. Overall, this research provides a powerful new tool for multimodal AI, with broad implications for both academia and industry, pushing the boundaries of what is possible in cross-modal content synthesis.
Deep Dive
Abstract
We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent. Existing multimodal generative models are largely LLM-based, which limits leveraging modality-specific generators and requires text-paired data for training. Recent diffusion- and flow-based any-to-any extensions take a different direction but still rely on text-aligned embeddings, fully-paired training, or matched-dimensionality deterministic mappings. MUNI rests on two complementary contributions, one architectural and one in the training objective. First, we extend latent diffusion to multimodal any-to-any generation end-to-end: instead of the standard two-stage recipe that precomputes a frozen latent space and then fits a prior over it, MUNI jointly trains modality-specific encoders, expressive decoders, and a single shared flow-based prior under one objective. Second, we identify that the standard aggregation rules of multimodal variational inference are insufficient once coupled with a learned prior and expressive decoders. A suitable shared latent must simultaneously satisfy coherence across generated modalities, predictive sufficiency of subset latents, and minimality of the latent content. We propose a routed training objective whose structural choices align the latent with these criteria and admit a minimal-sufficiency characterization in the realizable setting. Experiments on PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark show MUNI matching or exceeding the strongest baselines on conditional generation while opening its largest margins on unconditional coherence. Project page: https://muni-proj.github.io/.