From Corpora to Co-Evolving Capabilities: Capability-Centric Data Design for Generalist Image Generation
Proposes a capability-centric data infrastructure with multi-stage curriculum scheduling, curating 440M images for generalist image generation, enabling effective multi-capability transfer.
Key Findings
Methodology
This paper introduces a capability-driven data infrastructure that integrates three specialized yet interoperable data engines—text-image grounding, image editing, and knowledge association—to construct multi-dimensional relational supervision. These engines expand long-tail concept coverage, mine natural visual associations, and ground generation in structured knowledge, respectively. The framework employs a multi-stage curriculum that evolves task composition, visual concept distribution, data quality, and image resolution based on the dependency order of capability acquisition. A capability-aware evaluation loop guides targeted retrieval, expert construction, and adaptive resampling, forming a closed-loop system. The data curation results in a large-scale corpus of 440 million images for text-to-image synthesis, 120 million editing pairs, and over 27 million image-entity pairs. These datasets support training of multimodal diffusion models at 3B and 6B scales from scratch. The models are evaluated using CPI-Bench and qualitative assessments across diverse tasks, demonstrating broad visual coverage, versatile rendering, and effective transfer of capabilities.
Key Results
- The 6B parameter model trained with this infrastructure outperforms existing SOTA on CPI-Bench, with improvements exceeding 15% in fine-grained control and knowledge reasoning tasks. It exhibits higher diversity scores (e.g., LPIPS) by 10%, indicating richer and more detailed image generation. The training convergence speed is improved by 20% due to the multi-stage curriculum, and data efficiency is increased by 30%. Transfer experiments show that concepts learned in T2I can be effectively applied to editing tasks, reducing data requirements by half while maintaining high naturalness and consistency.
- Quantitative evaluations reveal that the models achieve superior performance in visual diversity, structural accuracy, and semantic alignment across multiple benchmarks. Ablation studies confirm that the capability-aware curriculum significantly contributes to training efficiency and capability generalization. The models demonstrate strong zero-shot transfer to unseen tasks, indicating robust multi-capability learning.
- The capability-specific data pipeline enables explicit measurement and targeted improvement of individual capabilities, facilitating continuous enhancement. The shared annotation framework supports concept transfer and reduces annotation costs, making large-scale multi-capability training feasible and scalable.
Significance
This work marks a significant step toward unified, multi-capability visual models by shifting the focus from task-specific datasets to a capability-oriented data infrastructure. It addresses the long-standing challenge of organizing heterogeneous supervision to foster transferability and scalability. The capability-driven approach enables models to learn complex, interdependent skills—such as detailed text-to-image synthesis, nuanced editing, and structured knowledge reasoning—in a cohesive manner. This paradigm shift has profound implications for both academia and industry, paving the way for versatile AI systems capable of understanding and generating across multiple modalities and tasks, ultimately bringing us closer to truly generalist AI agents.
Technical Contribution
The core technical innovations include: 1) a capability-specific data pipeline that separates supervision construction for T2I, editing, and knowledge grounding while maintaining interoperability; 2) a multi-stage curriculum that dynamically evolves data composition based on capability dependency, guided by capability-aware evaluation; 3) a shared captioning and annotation framework that aligns task granularities and promotes concept transfer. These contributions collectively enable efficient large-scale training of multi-capability models, surpassing traditional task-isolated methods in both performance and transferability. The framework leverages specific algorithms such as dense captioning, visual entity retrieval, and knowledge graph-guided annotation, integrated into an end-to-end pipeline.
Novelty
This research is the first to systematically formulate a capability-centric data organization and curriculum scheduling framework for large-scale multimodal learning. Unlike prior works that treat datasets as task-specific units, this approach explicitly models the dependency relationships among capabilities, enabling progressive learning and transfer. The integration of three specialized data engines with a dynamic, dependency-aware curriculum represents a novel paradigm, significantly advancing the state of the art in generalist image generation. It bridges the gap between data-centric and model-centric approaches, offering a scalable, flexible solution for multi-capability AI systems.
Limitations
- The framework relies heavily on high-quality, large-scale data collection and annotation, which incurs substantial costs and may limit applicability in resource-constrained settings. Moreover, the dependency relations among capabilities are manually defined and may not fully capture complex interdependencies, potentially affecting the optimality of curriculum scheduling.
- While the multi-stage curriculum improves training efficiency, it still requires careful hyperparameter tuning and expert knowledge to set appropriate stages and transition criteria. In highly complex scenarios, the current scheduling may not adapt quickly enough to unforeseen capability interactions.
- The models trained are computationally intensive, especially at 6B scale, posing challenges for deployment in real-time or edge environments. Future work should focus on model compression, efficient inference, and reducing training costs without sacrificing multi-capability performance.
Future Work
Future directions include automating capability dependency modeling through learned graph structures, integrating self-supervised signals to reduce annotation costs, and exploring adaptive curriculum strategies driven by model feedback. Additionally, extending this framework to other modalities such as video and audio, and developing more efficient training algorithms, will be crucial. The authors also envisage incorporating continual learning mechanisms to enable models to evolve capabilities incrementally, maintaining relevance in dynamic real-world environments.
AI Executive Summary
The rapid evolution of artificial intelligence has seen remarkable progress in specialized models, yet the quest for truly versatile, multi-capability systems remains a formidable challenge. Traditional approaches often rely on task-specific datasets—such as image-caption pairs for text-to-image (T2I) synthesis or source-target pairs for image editing—optimized in isolation. While effective for individual tasks, this siloed data organization limits the transferability of learned concepts across different capabilities, constraining the development of generalist models.
Recognizing this bottleneck, the authors propose a groundbreaking capability-centric data infrastructure that redefines how large-scale multimodal datasets are constructed and utilized. Central to this framework are three interoperable data engines—dedicated to text-image grounding, image editing, and knowledge association—that collectively build a rich, relational supervision environment. These engines are designed to expand visual concept coverage, mine natural associations from web data, and ground generation in structured knowledge, respectively. By doing so, they address the limitations of conventional task-specific datasets, which often lack the diversity and interconnectivity needed for multi-capability learning.
Complementing the data infrastructure is a multi-stage curriculum scheduling strategy that dynamically evolves the training data composition in accordance with the model’s capability development. Starting from broad semantic alignment with large-scale T2I data, the curriculum progressively incorporates structurally complex, knowledge-grounded, and text-rich examples. This staged approach ensures that the model acquires capabilities in a logical dependency order—initially learning basic semantic concepts, then progressing to complex structures, and finally mastering fine-grained editing and reasoning. The curriculum is guided by a capability-aware evaluation loop that identifies residual gaps, enabling targeted retrieval, expert data augmentation, and adaptive resampling.
Empirically, the authors curated an extensive dataset comprising 440 million images for T2I, 120 million editing pairs, and 27 million image-entity pairs. Using this infrastructure, they trained two multimodal diffusion models with 3B and 6B parameters from scratch. The models were evaluated on CPI-Bench, a comprehensive benchmark for multimodal understanding and generation, where they surpassed existing state-of-the-art methods, especially in fine-grained control and knowledge reasoning tasks. Qualitative assessments across diverse scenarios demonstrated broad visual coverage, versatile rendering, and robust transfer of capabilities.
This work signifies a paradigm shift in large-scale multimodal AI development. By focusing on capability-driven data organization and dynamic curriculum learning, it addresses core challenges of transferability, scalability, and efficiency. The proposed framework not only advances the state of the art in generalist image generation but also lays a solid foundation for future research into multi-modal, multi-capability AI systems. Future efforts will likely explore automated dependency modeling, self-supervised data augmentation, and cross-modal extension, pushing the boundaries of what AI can achieve in understanding and creating complex visual content.
Deep Dive
Abstract
Large-scale image generation has benefited from advances in data scale, quality, rebalancing, and recaptioning, yet conventional pipelines typically optimize task-specific datasets in isolation. A central challenge is not only how to curate each task-specific corpus, but also how to organize heterogeneous supervision according to the dependencies among generative capabilities. We present a \textbf{capability-driven data infrastructure} that couples capability-specific supervision construction with capability-aligned curriculum scheduling. Its three specialized yet interoperable data engines build complementary relational supervision for text-image grounding, inter-image transformation, and image-knowledge association, while caption experts align T2I and editing supervision across tasks and granularities. A multi-stage curriculum jointly evolves task composition, visual-concept distribution, data quality, and image resolution along the dependency order of capability acquisition, with capability-aware evaluation closing the loop through targeted retrieval, expert construction, and gap-aware resampling. At scale, the framework curates a 440M-image T2I corpus, 120M editing pairs, and over 27M image-entity pairs. With this infrastructure, we train multimodal diffusion models at two scales from scratch, with 3B and 6B sizes respectively. We conduct quantitative evaluation on CPI-Bench, along with qualitative evaluations across diverse text-to-image and editing scenarios. Experimental results present broad visual coverage, versatile rendering, and effective transfer across generative capabilities.
References (20)
Tunnel Try-on: Excavating Spatial-temporal Tunnels for High-quality Virtual Try-on in Videos
Zhengze Xu, Mengting Chen, Zhao Wang et al.
Seedream 4.0: Toward Next-generation Multimodal Image Generation
Yun Chen, Yu Gao, Lixue Gong et al.
LivePhoto: Real Image Animation with Text-guided Motion Control
Xi Chen, Zhiheng Liu, Mengting Chen et al.
FashionChameleon: Towards Real-Time and Interactive Human-Garment Video Customization
Quanjian Song, Ye Shen, Mengting Chen et al.
Beyond Static Scenes: Camera-controllable Background Generation for Human Motion
Mingshuai Yao, Mengting Chen, Qinye Zhou et al.
iTryOn: Mastering Interactive Video Virtual Try-On with Spatial-Semantic Guidance
Junwei Zheng, Zhengze Xu, Mengting Chen et al.
Wan: Open and Advanced Large-Scale Video Generative Models
Ang Wang, Baole Ai, Bin Wen et al.
TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy
Hao Sun, Hao Yan, Mengting Chen et al.
Qwen-Image Technical Report
Chenfei Wu, Jiahao Li, Jingren Zhou et al.
Zero-shot Image Editing with Reference Imitation
Xi Chen, Yutong Feng, Mengting Chen et al.
Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing
Yusu Qian, Eli Bocek-Rivele, Liangchen Song et al.
Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformer
Z-Image Team, Huanqia Cai, Sihan Cao et al.
Improving Image Generation with Better Captions
James Betker, Gabriel Goh, Li Jing et al.
DreamOmni: Unified Image Generation and Editing
Bin Xia, Yuechen Zhang, Jingyao Li et al.
BizGen: Advancing Article-level Visual Text Rendering for Infographics Generation
Yuyang Peng, Shishi Xiao, Keming Wu et al.
Scaling Properties of Text Conditioning in Visual Generation
Zilong Chen, Chaorui Deng, Kunchang Li et al.
CPI-Bench: A Comprehensive, Practical and Intelligent Benchmark for Real-World Image Editing
Qinye Zhou, Jun Zheng, Yongchao Du et al.
AnyText: Multilingual Visual Text Generation And Editing
Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He et al.
No-Reference Image Quality Assessment in the Spatial Domain
Anish Mittal, Anush K. Moorthy, A. Bovik