DuoGen: Towards General Purpose Interleaved Multimodal Generation
DuoGen framework enhances interleaved multimodal generation by data curation and architecture design, improving text quality and image consistency.
Key Findings
Methodology
DuoGen combines a pretrained multimodal large language model (MLLM) and a diffusion transformer (DiT) for video generation. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns the DiT using curated interleaved image-text sequences.
Key Results
- On CoMM and InterleavedBench benchmarks, DuoGen surpasses existing open-source models in text quality, image fidelity, and image-context alignment.
- DuoGen achieves state-of-the-art performance in text-to-image and image editing tasks, outperforming unified models like Bagel and OmniGen2.
- Significantly improves visual quality of instruction-tuning data with 298k interleaved conversation samples and 30k high-quality synthetic data.
Significance
DuoGen holds significant implications for academia and industry, addressing long-standing challenges of insufficient training data and limited base model capacity in existing interleaved generation models. Through systematic data curation and architecture design, DuoGen enhances the quality and consistency of text and image generation.
Technical Contribution
DuoGen offers substantial technical contributions by avoiding costly unimodal pretraining, allowing flexible base model selection, and achieving efficient interleaved generation through a decoupled training strategy.
Novelty
DuoGen is the first to systematically address data, training, and evaluation in general-purpose interleaved generation. Compared to existing visual chain-of-thought systems, DuoGen offers significant innovations in task coverage and generation quality.
Limitations
- In complex scenarios, generated images may lack detail and realism, possibly due to insufficient diversity in training data.
- Performance may degrade when handling extremely long text or image sequences.
Future Work
Future directions include expanding the diversity and scale of datasets, further optimizing the model's generation capabilities, and exploring more application scenarios.
AI Executive Summary
The DuoGen framework aims to tackle the challenges of insufficient training data and limited base model capacity in existing interleaved multimodal generation models. By combining a pretrained multimodal large language model and a diffusion transformer for video generation, DuoGen achieves efficient interleaved generation. Experimental results demonstrate that DuoGen surpasses existing open-source models in text quality, image fidelity, and image-context alignment.
The core technologies of DuoGen include a two-stage decoupled training strategy, first instruction-tuning the multimodal large language model, then aligning the diffusion transformer using curated interleaved image-text sequences. This approach avoids costly unimodal pretraining and allows flexible base model selection.
While DuoGen performs excellently across multiple benchmarks, generated images may lack detail and realism in complex scenarios. Future research directions include expanding the diversity and scale of datasets and exploring more application scenarios.
Deep Analysis
Background
Recent years have seen significant advances in multimodal generation models for image generation and text understanding. However, existing models often face limitations in handling interleaved multimodal generation tasks due to insufficient training data and limited base model capacity. The DuoGen framework addresses these issues through systematic data curation and architecture design.
Core Problem
Interleaved multimodal generation requires tightly coupled text and image outputs to support applications like step-by-step guides, visual planning, and interactive editing. Existing models often lack high-quality instruction-tuning data and flexible base model selection when handling these tasks.
Innovation
DuoGen's core innovations include: 1) Combining a pretrained multimodal large language model and a diffusion transformer for video generation, avoiding costly unimodal pretraining; 2) Implementing a two-stage decoupled training strategy, first instruction-tuning, then aligning the diffusion transformer.
Methodology
- �� Data Curation: Construct a large-scale high-quality instruction-tuning dataset combining multimodal conversations and synthetic examples.
- �� Architecture Design: Utilize pretrained multimodal large language model and diffusion transformer for flexible base model selection.
- �� Decoupled Training: First instruction-tune the MLLM, then align the DiT using interleaved image-text sequences.
Experiments
Experiments were conducted on CoMM and InterleavedBench benchmarks, covering diverse tasks and input formats. A new benchmark was constructed to test diverse everyday problems, evaluating the model's performance in text quality, image fidelity, and image-context alignment.
Results
DuoGen consistently outperforms previous open-source methods across all benchmarks, achieving significant improvements in text quality, image fidelity, and image-context alignment. Compared to models like Bagel and OmniGen2, DuoGen excels in text-to-image and image editing tasks.
Applications
DuoGen is suitable for applications requiring tightly coupled text and image outputs, such as step-by-step guides, visual planning, and interactive editing. Its flexible base model selection and efficient generation capabilities make it broadly applicable in academia and industry.
Limitations & Outlook
While DuoGen performs excellently across multiple benchmarks, generated images may lack detail and realism in complex scenarios. Additionally, performance may degrade when handling extremely long text or image sequences. Future research directions include expanding the diversity and scale of datasets and exploring more application scenarios.
Plain Language Accessible to non-experts
Imagine you're in a kitchen cooking. DuoGen is like a smart assistant that helps you prepare ingredients and cooking steps simultaneously. You tell the assistant what dish you want to make, and it helps you gather all the necessary ingredients and guides you step-by-step through the cooking process. The assistant not only provides detailed text instructions but also shows you pictures of each step, making it easier for you to understand. This way, even if you're a beginner, you can easily make delicious dishes.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a super cool game called DuoGen. This game lets you see both text and pictures at the same time, like reading an interactive novel. You can tell the game what you want, and it generates storylines and related pictures for you. For example, if you want to know how to make a delicious burger, DuoGen will tell you step-by-step how to do it and show you pictures of each step. Isn't that awesome?
Glossary
Multimodal Large Language Model (MLLM)
A pretrained model combining text and image understanding, capable of generating and understanding multimodal content.
Used in DuoGen for text generation and triggering image generation.
Diffusion Transformer (DiT)
A model for image generation, capable of producing high-quality image sequences under given conditions.
Used in DuoGen for generating interleaved image content.
Instruction Tuning
Fine-tuning a model with specific instruction data to improve its performance on particular tasks.
Used to optimize the MLLM's generation capabilities.
Interleaved Image-Text Sequence
A sequence combining text and images for training and evaluating multimodal generation models.
Used to align the DiT and MLLM's generation capabilities.
Video Generation
A technique for generating continuous image sequences, often used to simulate dynamic scenes.
Used as a pretraining task for the DiT in DuoGen.
Open Questions Unanswered questions from this research
- 1 How to improve the detail and realism of generated images in more complex scenarios?
- 2 How to further optimize the model's performance on extremely long text or image sequences?
Applications
Immediate Applications
Interactive Tutorials
DuoGen can be used to create interactive tutorials, providing step-by-step guidance and visual examples to help users better understand and complete tasks.
Long-term Vision
Intelligent Assistant
DuoGen can evolve into an intelligent assistant capable of providing real-time text and image support in various scenarios, enhancing user experience.
Abstract
Interleaved multimodal generation enables capabilities beyond unimodal generation models, such as step-by-step instructional guides, visual planning, and generating visual drafts for reasoning. However, the quality of existing interleaved generation models under general instructions remains limited by insufficient training data and base model capacity. We present DuoGen, a general-purpose interleaved generation framework that systematically addresses data curation, architecture design, and evaluation. On the data side, we build a large-scale, high-quality instruction-tuning dataset by combining multimodal conversations rewritten from curated raw websites, and diverse synthetic examples covering everyday scenarios. Architecturally, DuoGen leverages the strong visual understanding of a pretrained multimodal LLM and the visual generation capabilities of a diffusion transformer (DiT) pretrained on video generation, avoiding costly unimodal pretraining and enabling flexible base model selection. A two-stage decoupled strategy first instruction-tunes the MLLM, then aligns DiT with it using curated interleaved image-text sequences. Across public and newly proposed benchmarks, DuoGen outperforms prior open-source models in text quality, image fidelity, and image-context alignment, and also achieves state-of-the-art performance on text-to-image and image editing among unified generation models. Data and code will be released at https://research.nvidia.com/labs/dir/duogen/.