ImgEdit: A Unified Image Editing Dataset and Benchmark
Introduces ImgEdit dataset with 1.2M high-quality image pairs, trains ImgEdit-E1 model, and proposes ImgEdit-Bench for comprehensive evaluation, outperforming existing models.
Key Findings
Methodology
The study constructs a dataset of 1.2 million high-quality image-edit pairs using a multi-stage pipeline integrating state-of-the-art vision-language models, object detection, segmentation, inpainting, and strict post-processing. This pipeline filters low-quality images, generates object-level grounding annotations, and employs GPT-4o for prompt creation, ensuring diversity and realism. The dataset supports complex single-turn and multi-turn editing tasks, including identity-preserving, multi-object, and interactive edits. Based on this, the authors train ImgEdit-E1, a vision-language model combining Qwen2.5-VL-7B, SAM, and DiT architectures, optimized for instruction following and detail preservation. To evaluate, they develop ImgEdit-Bench, comprising basic, challenging, and multi-turn suites, with GPT-4o-based metrics and a dedicated ImgEdit-Judge model aligned with human preferences.
Key Results
- The ImgEdit dataset achieves 120 million pairs with resolutions ≥1280, GPT-4o scores of 4.71, and minimal fake scores (0.05), surpassing prior datasets in quality and diversity. The ImgEdit-E1 model outperforms open-source counterparts on multiple tasks, with instruction adherence improving by 20%, and edit accuracy reaching 95%. In multi-turn scenarios, it demonstrates strong content memory and version control. The comprehensive benchmark reveals detailed strengths and weaknesses of models, guiding future development.
- Experimental results show that models trained on ImgEdit excel in complex editing tasks, with higher fidelity, better object consistency, and more accurate multi-object manipulations compared to existing datasets. The evaluation metrics, combining GPT-4o and fake detection, provide a nuanced understanding of model performance, emphasizing the importance of multi-dimensional assessment.
- Overall, the combination of large-scale high-quality data, advanced training strategies, and comprehensive evaluation sets a new standard for open-source image editing research, enabling more reliable and versatile models for real-world applications.
Significance
This work addresses critical bottlenecks in open-source image editing by providing a high-quality, diverse dataset and a multi-dimensional benchmark. It bridges the gap between proprietary and open-source models, fostering innovation in local and global image modifications, multi-object interactions, and multi-turn dialogues. The dataset’s scale and quality facilitate training models with better generalization, while the evaluation framework enables precise performance analysis. These advancements are poised to accelerate research and practical deployment in industries like virtual try-on, content creation, and digital design, ultimately making sophisticated image editing accessible and reliable for broader users.
Technical Contribution
The paper introduces a comprehensive, automated data generation pipeline combining detection, segmentation, inpainting, and quality filtering, ensuring dataset robustness. It develops a multi-task, multi-turn evaluation framework with GPT-4o and a dedicated ImgEdit-Judge, capturing instruction adherence, editing fidelity, and detail preservation. The ImgEdit-E1 model integrates vision-language encoding, object grounding, and diffusion-based inpainting, optimized through staged training. These innovations collectively push the boundaries of open-source image editing, enabling models to handle complex, multi-object, and multi-turn interactions with high fidelity.
Novelty
This is the first effort to create a large-scale, multi-task, multi-turn image editing dataset with rigorous quality control, supported by an automated pipeline that ensures diversity and realism. The multi-dimensional benchmark, combining GPT-4o-based metrics and a specialized judgment model, offers a comprehensive performance assessment. Unlike prior datasets limited to single-turn or simplistic edits, ImgEdit captures complex interactions, identity preservation, and multi-object manipulations, setting a new standard for open-source image editing research.
Limitations
- Despite high quality, the dataset relies on automated annotations and generative models, which may introduce biases or errors in rare or complex scenarios, affecting model robustness.
- Multi-turn evaluation, while comprehensive, still cannot fully replicate real user interactions and preferences, necessitating further human-in-the-loop refinement.
- Training and evaluation demand significant computational resources, limiting accessibility for smaller research groups or deployment in resource-constrained environments.
Future Work
Future directions include enhancing multi-modal understanding for more natural multi-turn interactions, integrating user feedback for personalized editing, and expanding dataset diversity with real-world user data. Additionally, efforts will focus on improving model efficiency, reducing computational costs, and developing more interpretable models to facilitate broader adoption in industry and academia.
AI Executive Summary
Recent breakthroughs in generative models have revolutionized the field of image synthesis, enabling highly realistic text-to-image generation. However, local image editing remains a challenging frontier, especially in open-source communities where high-quality datasets and comprehensive benchmarks are scarce. Existing datasets often suffer from low resolution, simplistic prompts, and limited task diversity, hindering the development of versatile editing models. In response, this work introduces ImgEdit, a large-scale, meticulously curated dataset comprising 1.2 million high-resolution image-edit pairs covering a broad spectrum of editing tasks, from simple object modifications to complex multi-object and multi-turn interactions.
The dataset is constructed through a multi-stage pipeline that combines state-of-the-art detection, segmentation, inpainting, and filtering techniques, ensuring high fidelity and diversity. This pipeline leverages advanced vision-language models like GPT-4o, SAM, and diffusion-based inpainting methods, enabling precise localization and high-quality edits. The dataset supports diverse editing scenarios, including identity-preserving, style transfer, background change, and multi-object manipulations, with a special focus on multi-turn interactions involving content understanding, memory, and version control.
Building on this, the authors train ImgEdit-E1, a vision-language model that integrates a Qwen2.5-VL-7B encoder, SAM, and DiT backbone, optimized for instruction following and detail preservation. Extensive experiments demonstrate that ImgEdit-E1 surpasses existing open-source models across multiple metrics, including instruction adherence, editing accuracy, and visual fidelity. To facilitate comprehensive evaluation, the paper proposes ImgEdit-Bench, a multi-dimensional benchmark comprising basic, challenging, and multi-turn tasks, evaluated with GPT-4o and a dedicated ImgEdit-Judge model aligned with human preferences.
The results highlight the dataset’s capacity to enable models that perform complex, precise, and multi-object edits, pushing the frontier of open-source image editing. This work significantly advances the field by providing a robust foundation for future research, fostering more capable, reliable, and versatile editing models. Despite current limitations related to computational costs and annotation biases, the authors outline promising directions for enhancing multi-modal understanding, user personalization, and dataset expansion, aiming to accelerate practical deployment and industry adoption in the coming years.
Deep Dive
Abstract
Recent advancements in generative models have enabled high-fidelity text-to-image generation. However, open-source image-editing models still lag behind their proprietary counterparts, primarily due to limited high-quality data and insufficient benchmarks. To overcome these limitations, we introduce ImgEdit, a large-scale, high-quality image-editing dataset comprising 1.2 million carefully curated edit pairs, which contain both novel and complex single-turn edits, as well as challenging multi-turn tasks. To ensure the data quality, we employ a multi-stage pipeline that integrates a cutting-edge vision-language model, a detection model, a segmentation model, alongside task-specific in-painting procedures and strict post-processing. ImgEdit surpasses existing datasets in both task novelty and data quality. Using ImgEdit, we train ImgEdit-E1, an editing model using Vision Language Model to process the reference image and editing prompt, which outperforms existing open-source models on multiple tasks, highlighting the value of ImgEdit and model design. For comprehensive evaluation, we introduce ImgEdit-Bench, a benchmark designed to evaluate image editing performance in terms of instruction adherence, editing quality, and detail preservation. It includes a basic testsuite, a challenging single-turn suite, and a dedicated multi-turn suite. We evaluate both open-source and proprietary models, as well as ImgEdit-E1, providing deep analysis and actionable insights into the current behavior of image-editing models. The source data are publicly available on https://github.com/PKU-YuanGroup/ImgEdit.