VideoBooth: Diffusion-based Video Generation with Image Prompts
VideoBooth enables feed-forward image-prompted video generation, reaching 65.0979 DINO and 74.7971 CLIP-Image.
Key Findings
Methodology
VideoBooth takes an image prompt I and text prompt T. At the coarse level, a CLIP Image Encoder extracts visual features, and MLPs map them into the text-embedding space, replacing the target-subject tokens. At the fine level, the image is encoded by the Stable Diffusion VAE into multi-scale latents and injected into a video diffusion U-Net's Cross-Frame Attention as extra keys and values. The first frame absorbs image details, and its updated values are propagated to later frames.
Key Results
- On 650 WebVid-10M test pairs, VideoBooth obtains 74.7971 CLIP-Image, above ELITE's 73.7518, DreamBooth's 71.2078, and Textual Inversion's 69.7995. Its DINO score is 65.0979, exceeding 58.9522, 52.9661, and 45.3143 for the same baselines.
- Its CLIP-Text score is 30.0967, comparable to ELITE's 30.0881 and Textual Inversion's 29.9749, although below DreamBooth's 30.6877. The result indicates stronger reference-subject fidelity without sacrificing text conditioning.
- Coarse-to-fine training is essential. The ablation reports that joint training lets fine attention injection leak overly strong visual cues, causing the coarse encoder to learn meaningless representations and weakening inference-time control.
Significance
The paper moves personalized video generation from describing a subject with words to specifying appearance directly with one reference image. This addresses the persistent difficulty of enumerating color, texture, identity, and local structure in text, while avoiding per-subject inference-time optimization used by DreamBooth-like systems. Academically, it defines a task, dataset construction procedure, and evaluation protocol for image-prompted video. Industrially, a single feed-forward model is better suited to rapid advertising, social-media, and e-commerce production.
Technical Contribution
The main contribution is a complementary coarse-to-fine control architecture. CLIP image features provide high-level semantics for the text condition, while Attention Injection preserves spatial layout and multi-scale texture. Crucially, the reference latent is no longer clean: it follows the diffusion forward process, xI_t=√α_t xI_0+√(1−α_t)ε, to match intermediate noise levels. Image keys and values first update the initial frame, whose values then condition subsequent frames, coupling appearance fidelity with temporal consistency.
Novelty
The authors present VideoBooth as the first systematic study of image-prompted video generation without inference-time fine-tuning. Unlike ELITE-style approaches that mainly compress an image into text-like tokens, VideoBooth combines global semantic conditioning with spatially preserved, multi-scale latent injection, then propagates the refined first-frame representation through cross-frame attention.
Limitations
- The data pipeline depends on Grounded-SAM masks extracted from the first WebVid frame, so segmentation errors, occlusions, background leakage, and selection bias can directly affect training and evaluation.
- Only 48,724 pairs were retained from a 2.5-million WebVid subset, with filtering centered on nine moving-object categories. Generalization to many subjects, articulated objects, multiple entities, and long videos is not established.
- The paper reports proxy similarity metrics but not inference speed, memory cost, human preference, or systematic long-horizon identity-drift measurements.
Future Work
The authors plan to process the full WebVid collection and enlarge the VideoBooth dataset. Important extensions include broader categories, multiple subjects, complex backgrounds, longer clips, stronger segmentation and tracking, and explicit motion control. The field also needs human-preference studies, creator-task evaluations, safety and copyright protocols, and computational-cost reporting beyond CLIP and DINO.
AI Executive Summary
Text-to-video models can produce plausible scenes, but text is a poor instrument for specifying the exact appearance of one individual subject. A prompt such as “a brown-and-white dog” does not enumerate every marking, texture, or facial cue; even elaborate descriptions may yield a dog that is semantically correct but visually wrong. Textual Inversion and DreamBooth improve personalization, yet commonly require subject-specific optimization or fine-tuning, limiting rapid creation.
VideoBooth addresses this gap with a feed-forward framework that combines image and text prompts. Its coarse branch uses a CLIP Image Encoder and MLPs to map a reference image into text-embedding space, replacing the target-subject tokens. Its fine branch encodes the image with the Stable Diffusion VAE and injects multi-scale noisy latents into Cross-Frame Attention inside a video diffusion U-Net. The first frame is refined with image keys and values; its updated values are then propagated to later frames. Training proceeds from coarse to fine because joint training makes the fine branch dominate and destabilizes the coarse representation.
On 650 held-out WebVid-10M pairs, VideoBooth achieves 74.7971 CLIP-Image and 65.0979 DINO, the best scores among the reported methods; relative to ELITE, these are gains of 1.0453 and 6.1457. CLIP-Text reaches 30.0967, essentially matching ELITE. The training resource contains 48,724 pairs filtered from a 2.5-million WebVid subset. The results establish a practical direction for customized video, although limited categories, short clips, proxy metrics, and segmentation-derived prompts leave substantial room for broader validation.
Deep Analysis
Background
Stable Diffusion accelerated text-to-image synthesis, while Make-A-Video, Video LDM, ModelScope, and related systems introduced temporal modeling for text-to-video generation. These models can express scenes and actions, but text poorly specifies a particular subject. Textual Inversion and DreamBooth personalize images through optimization, whereas ELITE uses encoder-based visual embeddings. Directly transferring such strategies to video can produce identity mismatch, flicker, or unnatural motion.
Core Problem
Given an image prompt I and text prompt T, the system must generate a moving video containing the specified subject. It must preserve fine color, texture, shape, and identity across frames; reconcile a clean reference image with noisy diffusion states; and maintain temporal coherence. The practical constraint is equally important: a new subject should not require inference-time optimization or model fine-tuning.
Innovation
- �� Coarse CLIP-MLP conditioning maps global visual semantics into the text space and replaces target-subject tokens.
- �� Fine Attention Injection appends multi-scale image latents as extra keys and values in Cross-Frame Attention.
- �� First-frame propagation updates the first frame with reference details and uses its values to condition later frames.
- �� Coarse-to-fine training first establishes subject generation, then learns detail injection, preventing signal competition between branches.
Methodology
- �� Input: image prompt I and text prompt T.
- �� Coarse encoding: compute fV=CLIPI(I), then fI=F(fV); replace the target subject's n text tokens with fI.
- �� Backbone: use a pretrained video diffusion U-Net with inflated 3D convolutions, Cross-Frame Attention, and Temporal Attention.
- �� Latent alignment: encode I as xI0 through the VAE and form xIt=√αtxI0+√(1−αt)ε.
- �� Fine injection: update the first-frame values using concatenated [KI,K0] and [VI,V0]; update later frames using original keys and the refined first-frame values.
- �� Multi-scale control: feed resolution-matched image latents into different U-Net attention layers.
Experiments
The dataset is derived from WebVid. From a 2.5-million subset, spaCy extracts noun chunks from captions, Grounded-SAM segments matching subjects in the first frame, and filtering removes very small, nearly full-frame, or non-moving objects. This yields 48,724 training pairs; the held-out benchmark contains 650 pairs from WebVid-10M. Baselines are adapted Textual Inversion, DreamBooth, and retrained ELITE. Metrics are frame-averaged CLIP-Text, CLIP-Image, and ViT-S/16 DINO similarity. Ablations test the coarse-to-fine training schedule.
Results
VideoBooth reaches 74.7971 CLIP-Image versus ELITE's 73.7518 and 65.0979 DINO versus 58.9522, a 6.1457-point DINO gain. Its CLIP-Text score is 30.0967, close to ELITE's 30.0881. DreamBooth leads text alignment at 30.6877, but the paper notes that its optimized token can increase textual relevance while weakening reference-image fidelity. Qualitative comparisons show stronger preservation of subject appearance.
Applications
The framework can generate advertisements in which a specified product or mascot performs a textual action, personalized e-commerce clips, social-media videos, and early film or educational visualization. A user needs only one reference image and a prompt, rather than subject-specific training. Deployment still requires reliable segmentation, copyright and likeness controls, safety filtering, provenance labels, and evaluation beyond embedding similarity.
Limitations & Outlook
Performance depends on accurate subject masks and suitable video data; small, occluded, cluttered, or overlapping subjects may fail. The training categories are narrow, and long videos, rapid motion, shot changes, and complex interactions are not systematically tested. CLIP and DINO are useful proxies but do not fully measure motion realism or user satisfaction. Future systems should add tracking, controllable motion, broader data, human evaluation, and transparent speed and memory measurements.
Plain Language Accessible to non-experts
Imagine a professional film studio that receives one photograph of your dog and the instruction, “Make it run through a meadow.” If the director hears only the words “brown-and-white dog,” they can invent a dog with the wrong markings. If they merely scan the photograph into one tiny label, they may remember the general animal but forget the ear shape, fur pattern, and face.
VideoBooth uses two passes. The first pass is like an assistant memorizing the dog’s identity and overall appearance. The second pass studies the photograph at several zoom levels, preserving both the body outline and small visual details. The studio first makes the opening shot match the photograph, then gives that corrected opening shot to the later shots as a reference. The dog can move, yet still look like the same dog.
The system also makes the photograph progressively noisier so it speaks the same “language” as the video-making process. In tests, it scored 65.0979 on DINO, above ELITE’s 58.9522, and achieved the highest CLIP-Image score of 74.7971. It can still struggle with occlusion, clutter, or long scenes, but it makes one-image-plus-one-sentence video creation substantially more practical.
ELI14 Explained like you're 14
Think about creating a game character from one photo of your pet. You upload the picture and type, “Let it run across a sunny field.” A normal generator might create a dog with different colors or a totally different face, because words cannot list every stripe, curl, and tiny marking.
VideoBooth gives the AI two pairs of glasses. The first pair helps it remember who the subject is: maybe a brown-and-white dog. The second pair lets it zoom in on the ears, fur, face, and body shape. It fixes the first video frame so it resembles the photo, then passes that visual memory to the next frames. So when the dog runs, it does not become a new dog every second!
The system also changes the photo to match the noisy stage used while making the video. That helps the reference and the video process communicate. Training happens in two steps: first learn the big picture, then learn tiny details. Otherwise, the AI might copy details without understanding the subject as a whole.
On 650 test examples, VideoBooth got 65.0979 DINO and 74.7971 CLIP-Image, beating the reported alternatives for visual matching. It is not magic: blocked subjects, crowded scenes, and very long videos can still confuse it. But it could help make ads, game clips, pet videos, and personalized social posts much faster!
Glossary
Image Prompt
A reference image supplied by the user to specify a subject's appearance. It provides visual attributes that text often cannot enumerate precisely.
VideoBooth combines image prompt I with text prompt T for video generation.
Coarse-to-Fine Embedding
A strategy that first encodes global semantics and then injects localized spatial detail. The coarse branch stabilizes identity, while the fine branch restores texture and structure.
VideoBooth implements the two stages with CLIP-MLP conditioning and Attention Injection.
Cross-Frame Attention
An attention mechanism through which one video frame references features from the first or preceding frame. It reduces identity drift and temporal flicker.
The paper adds image keys and values to this module.
DINO Similarity
A similarity measure based on DINO visual features between a reference subject and generated frames. It is intended to capture instance-level differences more sensitively than CLIP alone.
VideoBooth obtains a DINO score of 65.0979.
Latent Diffusion
A diffusion-generation paradigm that performs denoising in a compressed latent space rather than directly on pixels. This improves computational efficiency.
The reference image is encoded by the Stable Diffusion VAE before attention injection.
Open Questions Unanswered questions from this research
- 1 Long-horizon identity stability remains unclear. The experiments focus on short WebVid clips and frame-averaged metrics; tracking, shot changes, and motion-intensity stratification require dedicated evaluation.
- 2 Grounded-SAM-derived masks may introduce bias and noise. More work is needed on occlusion, precise segmentation, multiple subjects, and visually similar identities.
- 3 CLIP and DINO do not fully measure motion naturalness or user preference. Human studies, physical plausibility tests, and real creator workflows are still needed.
Applications
Immediate Applications
E-commerce and advertising clips
Brands can provide one product or mascot image and specify scenes or actions in text to create many promotional variants. Clear references and rights clearance are prerequisites; the expected benefit is reduced subject-specific modeling and shooting cost.
Personalized social content
Users can upload a pet, character, or collectible image and request actions such as “running in snow.” Platforms should add likeness and copyright checks, watermarks, safety filters, and provenance records before distribution.
Long-term Vision
Controllable film and virtual-character production
A reference character could eventually drive continuous scenes, camera plans, and action variations for storyboarding, games, and virtual influencers. Major obstacles are long-term identity, multi-character interaction, repeatable motion, rights management, and reliable temporal control.
Abstract
Text-driven video generation witnesses rapid progress. However, merely using text prompts is not enough to depict the desired subject appearance that accurately aligns with users' intents, especially for customized content creation. In this paper, we study the task of video generation with image prompts, which provide more accurate and direct content control beyond the text prompts. Specifically, we propose a feed-forward framework VideoBooth, with two dedicated designs: 1) We propose to embed image prompts in a coarse-to-fine manner. Coarse visual embeddings from image encoder provide high-level encodings of image prompts, while fine visual embeddings from the proposed attention injection module provide multi-scale and detailed encoding of image prompts. These two complementary embeddings can faithfully capture the desired appearance. 2) In the attention injection module at fine level, multi-scale image prompts are fed into different cross-frame attention layers as additional keys and values. This extra spatial information refines the details in the first frame and then it is propagated to the remaining frames, which maintains temporal consistency. Extensive experiments demonstrate that VideoBooth achieves state-of-the-art performance in generating customized high-quality videos with subjects specified in image prompts. Notably, VideoBooth is a generalizable framework where a single model works for a wide range of image prompts with feed-forward pass.