DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models

Recent years have seen rapid progress in diffusion-based generative models, such as Rombach et al.'s Stable Diffusion and Ramesh et al.'s DALL·E 2, which leverage large-scale image-text datasets like LAION-2B. These models excel at producing high-fidelity images conditioned on textual prompts, transforming creative industries and content creation. Prior work focused on model architecture improvements, training efficiency, and dataset scaling, but lacked comprehensive analysis of user prompts and their influence on output quality. As user engagement with these models grows, understanding prompt design becomes critical. However, existing datasets are limited in size and scope, often synthetic or curated, restricting insights into real-world prompt usage. This gap hinders systematic improvements in prompt engineering, safety, and bias mitigation. Our work addresses this by collecting authentic user prompts and images from a public Discord community, enabling large-scale analysis of prompt styles, hyperparameters, and potential misuse scenarios.

Despite the impressive capabilities of diffusion models, users face significant hurdles in crafting effective prompts due to trial-and-error processes and lack of guidance. The variability in prompt styles, language, and hyperparameters leads to inconsistent outputs, especially when models generate biased, misleading, or harmful content. Moreover, current datasets do not reflect real user interactions, limiting the understanding of prompt-driven errors and misuse. This impairs efforts to develop robust, controllable, and safe generative systems. The core challenge lies in systematically analyzing the complex interplay between prompt features, hyperparameters, and generated content at scale, while identifying potential risks such as misinformation, non-consensual imagery, and bias.

This study introduces several key innovations: 1) Construction of DiffusionDB, a 6.5TB dataset with 14 million images and 1.8 million prompts from real users, capturing diverse styles and hyperparameters. 2) Application of syntactic parsing (SpaCy) and semantic embedding (CLIP + UMAP) to analyze prompt structures and semantic distributions, revealing key patterns and biases. 3) Linking hyperparameters like CFG scale, steps, and seed to generation errors via regression analysis, identifying conditions leading to poor outputs and misinformation risks. 4) Development of a comprehensive content filtering pipeline using state-of-the-art NSFW classifiers, enabling safer dataset sharing. These innovations facilitate large-scale, data-driven insights into prompt design and model behavior, bridging the gap between user interaction and technical understanding.

• �� Data collection: Scrape images and prompts from Stable Diffusion Discord using DiscordChatExporter, ensuring compliance with community rules.
• �� Metadata extraction: Parse HTML with BeautifulSoup, associate images with prompts, hyperparameters, user IDs, timestamps.
• �� Content filtering: Apply multilingual toxicity models and EfficientNet classifiers to detect NSFW content, assign scores.
• �� Data organization: Store images with UUID filenames, organize into subfolders, create JSON metadata files.
• �� Syntax analysis: Use SpaCy to extract named entities and noun phrases, analyze prompt structure.
• �� Semantic analysis: Encode prompts and images with CLIP, reduce dimensions with UMAP, visualize distributions.
• �� Error detection: Compute CLIP cosine distances between prompts and images, identify outliers, analyze hyperparameter effects on errors.

Experiments involved statistical analysis of prompt length, language, syntactic and semantic features, and hyperparameter effects. CLIP embedding distances identified mismatched prompt-image pairs. Regression models linked hyperparameter deviations to errors. Content filtering validated NSFW detection accuracy. The dataset's diversity allowed cross-category analysis, revealing style patterns and bias tendencies. These experiments provided comprehensive insights into prompt design, model errors, and misuse scenarios, guiding future improvements in prompt engineering and safety mechanisms.

The dataset's scale and diversity enabled detailed analysis of prompt styles, hyperparameter impacts, and content biases. Common prompt patterns like ‘trending on artstation’ correlated with higher error rates. Deviations in CFG scale and steps significantly increased the likelihood of poor outputs, especially with negative CFG values or small step counts. Semantic embedding visualization uncovered distinct clusters for art and photography prompts, with biases in generated images of faces and political figures. The analysis also revealed potential misuse, including prompts that generate misinformation or non-consensual content, emphasizing the need for safer deployment.

DiffusionDB supports prompt auto-completion, content moderation, and bias detection tools. It aids industry applications like content creation platforms, enabling users to craft better prompts and avoid harmful outputs. Academic research can leverage the dataset for explainability, bias mitigation, and safety studies. The insights can inform the design of user interfaces that guide prompt formulation, reducing trial-and-error and enhancing controllability. Overall, it fosters responsible AI development in image synthesis.

The dataset is platform-specific, primarily derived from Discord, which may introduce sampling bias. NSFW classifiers, though effective, are not infallible, risking false negatives. The analysis based on CLIP distances may overlook other failure modes. Computational costs for processing such large data are high, limiting real-time applications. Future work should incorporate broader data sources, multimodal content detection, and more robust bias mitigation strategies.