In-Context Learning Unlocked for Diffusion Models

The evolution of generative models has seen diffusion models like DDPM and LDM achieve remarkable success in high-fidelity image synthesis. Simultaneously, large-scale language models such as GPT-3 and GPT-4 exhibit strong contextual understanding and zero-shot capabilities. Recent efforts have integrated these modalities, leading to multimodal models like CLIP and Flamingo. However, most diffusion models remain task-specific, lacking flexible multi-task and zero-shot abilities. Addressing this gap, recent research explores multi-task training and prompt engineering, but challenges remain in effectively combining diverse tasks and ensuring generalization.

Current diffusion models are predominantly trained for single tasks, limiting their adaptability to new or complex scenarios. Designing prompts that effectively encode multiple vision-language tasks is non-trivial, especially when aiming for models that can understand and perform unseen tasks without retraining. Moreover, existing models struggle with generalization, often requiring task-specific fine-tuning. These limitations hinder the deployment of versatile, intelligent content generation systems capable of handling diverse real-world applications.

This work introduces a multi-task, multimodal prompt framework that unifies six vision-language tasks within a single diffusion model. Key innovations include a novel prompt design combining image pairs and text guidance, a ControlNet-based architecture for multimodal conditioning, and joint training across tasks to learn inter-task relationships. These innovations enable the model to understand context, transfer knowledge to unseen tasks, and perform zero-shot inference, significantly advancing the flexibility and generalization of diffusion models.

• �� Design a multimodal prompt format: combining paired images, text guidance, and query images in a unified input.
• �� Build Prompt Diffusion: extend ControlNet with concatenated image embeddings and CLIP text encodings, integrating via cross-attention layers.
• �� Multi-task training: randomly sample six tasks per batch, jointly optimize model parameters, and learn task relationships.
• �� Use classifier-free guidance during training, randomly dropping text guidance to improve robustness.
• �� Validate via extensive experiments on datasets with 310k image-caption pairs, depth, HED, and segmentation maps, assessing zero-shot and few-shot performance across tasks.

The dataset includes image-caption pairs, depth, HED, and segmentation maps, with data augmentation via Midas and Uniformer. Models are trained on 8×A100 GPUs for 5000-20000 steps, with evaluation metrics including FID and RMSE. The experiments compare Prompt Diffusion against baseline models like ControlNet fine-tuned on individual tasks, analyzing generalization to unseen tasks such as scribble and edge generation. Ablation studies examine prompt design components, multi-task training effects, and model robustness across scenarios.

Prompt Diffusion achieves 85.4% zero-shot FID on trained tasks, outperforming single-task models. It generalizes effectively to unseen tasks, producing natural images with structural consistency. In style transfer and pixel misalignment tasks, it generates high-quality outputs without additional training. The model demonstrates superior flexibility, supporting complex content transformations guided solely by prompts, validating the effectiveness of multi-task joint training and multimodal prompting.

The framework can be applied in automated content creation, virtual environment design, and intelligent editing tools. It enables users to generate diverse visual content with minimal examples and textual instructions, reducing reliance on task-specific models. Long-term, it could underpin interactive AI systems capable of understanding and executing a wide range of visual-language tasks, transforming industries like entertainment, advertising, and education.

High computational costs limit real-time deployment, requiring extensive GPU resources. The model’s performance may decline in extremely high-resolution or highly detailed scenarios. Generalization to highly specialized domains remains limited, necessitating further dataset expansion and prompt optimization. Future work should focus on improving efficiency, robustness, and scalability.