VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Models

Recent years have seen rapid advances in text-to-video diffusion models such as Pika, Text2Video-Zero, VideoCraft2, and ModelScope, which leverage diffusion mechanisms to generate dynamic videos from textual prompts. These models have revolutionized content creation, enabling applications in entertainment, virtual reality, and advertising. However, progress is limited by the scarcity of large, diverse, and real-user prompt datasets that accurately reflect practical usage scenarios. Existing datasets like WebVid-10M and HDVILA-100M focus on caption-video pairs but lack the richness of user-generated prompts tailored for dynamic video content. Prompt engineering has become crucial for improving quality, safety, and relevance, yet there is no comprehensive resource to support systematic research. This gap hampers the development of more robust, efficient, and safe models, especially in real-world applications.

The core challenge in text-to-video generation lies in capturing the complexity and diversity of user prompts, which often involve dynamic actions, long descriptions, and contextual details. Current datasets are limited in scale and scope, restricting model generalization and safety. Moreover, the high computational cost of video synthesis makes large-scale data collection difficult. There is an urgent need for a comprehensive, large-scale prompt dataset that reflects real user behavior, supports diverse content, and facilitates evaluation and safety analysis. Without such a resource, models risk overfitting to narrow domains, producing biased or unsafe content, and failing to meet industry standards for quality and reliability.

VidProM introduces several key innovations: 1) It systematically collects prompts from real users via Pika Discord, capturing authentic, dynamic descriptions; 2) It employs OpenAI text-embedding-3-large, supporting long prompts up to 8192 tokens, enabling detailed scene descriptions; 3) It integrates videos generated by four state-of-the-art diffusion models, ensuring content diversity; 4) It applies semantic uniqueness filtering based on cosine similarity, enhancing content variety and reducing redundancy. These innovations collectively address limitations of existing datasets, particularly in capturing the dynamic, complex nature of video prompts, and provide a rich resource for training, evaluation, and safety research.

• �� Data collection: Extract prompts from Pika Discord channels using regex, filter duplicates, assign UUIDs and timestamps.
• �� Embedding: Use OpenAI text-embedding-3-large to convert prompts into high-dimensional vectors, supporting up to 8192 tokens.
• �� NSFW scoring: Apply Detoxify to assign toxicity, obscenity, and other safety scores, filtering potentially unsafe prompts.
• �� Video generation: Distribute across 10 servers, run four diffusion models (Pika, Text2Video-Zero, VideoCraft2, ModelScope), generate videos.
• �� Semantic filtering: Calculate cosine similarity between prompt embeddings, remove semantically similar prompts (similarity > 0.8).
• �� Data integration: Combine prompts, videos, embeddings, NSFW scores into the VidProM dataset, ensuring diversity and safety.

The dataset's validity was tested through semantic analysis, visualization with t-SNE and WizMap, and model evaluation experiments. Prompts were analyzed for length, complexity, and thematic distribution, showing significant differences from DiffusionDB. Model performance was benchmarked using metrics like FID and CLIPScore, demonstrating improved generalization on real-user prompts. Safety filtering effectiveness was validated via NSFW scores, ensuring content appropriateness. The experiments confirmed that VidProM enhances model robustness, diversity, and safety, providing a realistic benchmark for future research.

VidProM's 16.72 million prompts and 6.69 million videos vastly expand the scope of prompt resources. The prompts are longer, with many exceeding 70 words, capturing detailed dynamic scenes. Semantic analysis reveals distinct distribution patterns, emphasizing human activities and complex actions. Models trained on VidProM outperform baselines on evaluation metrics, demonstrating better generalization to real-world prompts. The dataset's diversity facilitates research in prompt engineering, safety, and efficient generation, marking a significant step forward in the field.

VidProM supports model evaluation, prompt optimization, safety research, and content moderation. It enables industry applications such as personalized content creation, virtual reality scene synthesis, and automated video production. The dataset also aids in developing safer, more reliable models for commercial deployment, ensuring content appropriateness and copyright compliance. Long-term, it can foster innovations in real-time, adaptive video synthesis, transforming entertainment, education, and communication industries.

Data collection is platform-specific, potentially introducing bias and limiting generalizability. Video generation costs are high, restricting dataset expansion. Generated videos reflect underlying model biases, raising safety and fairness concerns. Future work should focus on broadening data sources, optimizing computational efficiency, and enhancing content safety measures to address these issues.