GuardPaint:SpeculativeSafetyDecodingforText-to-ImageGeneration
GuardPaint uses speculative decoding to repair unsafe regions during diffusion, reducing attack success rates while preserving image quality.
Key Findings
Methodology
GuardPaint introduces a speculative decoding framework where a lightweight auditor monitors intermediate images, identifies unsafe regions, and triggers localized repairs using a policy-aligned inpainter. Repairs are selected via a guarded tournament, ensuring compliance with safety policies while preserving prompt fidelity and perceptual quality.
Key Results
- On SD 1.5, DACA attack success rate dropped from 8.25% to 1.94%, SneakPrompt from 7.00% to 1.79%.
- On FLUX.1-dev, MMA attack success rate reduced from 4.25% to 2.44%.
- Post-repair image fidelity and quality remained within ±0.05 of the baseline, showing no significant degradation.
Significance
This study addresses safety challenges in diffusion models by providing a modular solution that intervenes during the generation process without modifying the base model. It prevents unsafe content generation and offers semantically coherent alternatives, applicable across multiple architectures.
Technical Contribution
Introduced a decoding-time safety layer combining auditing, localized repair, and tournament-based selection, enabling trajectory-level safety intervention without requiring model retraining or global distribution changes.
Novelty
GuardPaint is the first framework to perform localized safety interventions during the diffusion trajectory, offering policy-compliant visual alternatives instead of outright refusals.
Limitations
- The inpainter is based solely on SD 1.5, potentially limiting stylistic fidelity for higher-capacity models.
- The auditor is trained on only three categories (safe, nudity, violence), lacking coverage for subtler policy violations.
Future Work
Future work could expand the auditor's taxonomy to detect more nuanced policy violations and develop native inpainters for diverse architectures.
AI Executive Summary
Text-to-image diffusion models excel at generating high-quality visuals but face critical safety challenges: adversarial prompts can lead to the creation of explicit or violent content. Existing safeguards focus on pre-generation prompt filtering or post-generation image classification, leaving the diffusion process unprotected.
GuardPaint proposes a speculative decoding framework that intervenes during the diffusion trajectory. A lightweight auditor monitors intermediate images, identifies unsafe regions, and triggers localized repairs. Policy-aligned inpainting candidates are generated and ranked through a guarded tournament, ensuring compliance with safety policies while preserving prompt fidelity. Experiments show significant reductions in attack success rates with minimal impact on image quality.
The framework is plug-and-play across architectures like SD 1.5, SDXL, and FLUX.1-dev without modifying base model weights. Future directions include expanding the auditor's taxonomy and developing native inpainters for diverse architectures.
Deep Analysis
Background
Diffusion models have revolutionized image generation, with notable works like Stable Diffusion and DALL-E. However, their openness and controllability make them vulnerable to adversarial prompts, leading to unsafe outputs such as explicit or violent images.
Core Problem
Existing safeguards focus on prompt filtering or post-generation classification, failing to intervene during the diffusion process. This results in either outright refusals or an inability to provide safe alternatives.
Innovation
GuardPaint introduces speculative decoding for trajectory-level safety intervention. Innovations include a lightweight auditor, policy-aligned inpainter, and guarded tournament mechanism, avoiding model retraining or global distribution changes.
Methodology
- �� Auditor monitors intermediate images to identify unsafe regions.
- �� Inpainter generates policy-compliant repair candidates.
- �� Guarded tournament selects repairs that meet safety, fidelity, and quality criteria.
- �� Repairs are reinserted into the diffusion trajectory using model-specific equations.
Experiments
Experiments use the JailBreakDiffBench protocol to evaluate five attack types (SneakPrompt, MMA, etc.) across four architectures. Metrics include attack success rate (ASR), image fidelity, and quality.
Results
GuardPaint significantly reduced ASR, e.g., DACA dropped from 8.25% to 1.94% on SD 1.5. Repairs maintained fidelity and quality within ±0.05 of the baseline.
Applications
Applicable to public generation systems and creative tools, providing semantically coherent alternatives for unsafe content generation.
Limitations & Outlook
The inpainter is based on SD 1.5, limiting stylistic fidelity for higher-capacity models. The auditor's taxonomy is coarse, unable to detect nuanced policy violations.
Plain Language Accessible to non-experts
Imagine GuardPaint as a smart assistant in a kitchen. During cooking, it checks each dish in real-time. If it finds a dish that's too salty or unhealthy, it adjusts only that dish without redoing the entire meal. This ensures the final meal is both delicious and healthy.
ELI14 Explained like you're 14
Think of GuardPaint like a gaming system that watches your moves. If you're about to trigger a dangerous trap, it quietly fixes the problem without ruining your game. In diffusion models, it fixes unsafe parts of an image while keeping the rest intact.
Glossary
Diffusion Model
A generative model that creates high-quality images by iteratively denoising. Used for text-to-image tasks.
Central to the paper's framework.
Speculative Decoding
A decoding method that improves safety by auditing and repairing content in real-time.
Core mechanism of GuardPaint.
Auditor
A module that detects unsafe regions during the diffusion process.
Monitors intermediate images.
Inpainter
A module that repairs unsafe regions to generate policy-compliant alternatives.
Produces localized repairs.
Guarded Tournament
A selection mechanism that accepts only repairs meeting safety and fidelity criteria.
Ensures repair quality.
Open Questions Unanswered questions from this research
- 1 How can auditors detect more nuanced policy violations?
- 2 How can native inpainters be developed for diverse architectures?
Applications
Immediate Applications
Content Moderation
Real-time detection and repair of unsafe content for generation platforms.
Creative Tools
Helps artists generate high-quality images compliant with policies.
Long-term Vision
Multimodal Safety
Extend to video, audio, and other generation tasks for comprehensive safety.
Abstract
Text-to-image (T2I) diffusion models offer powerful visual generation, but their controllability creates a critical safety challenge: adversarial prompts can steer the denoising trajectory toward policy-violating content such as explicit nudity or graphic violence. Existing safeguards mostly act before generation through prompt filtering or after generation through image classification, leaving the diffusion process itself unguarded and often yielding only refusal rather than safe visual repair. We introduce GuardPaint, a speculative decoding framework for safe T2I generation that intervenes inside the diffusion trajectory without modifying the base model. A lightweight auditor monitors intermediate images, localizes unsafe regions, and triggers surgical inpainting repair only where needed. Candidate repairs are generated by a policy-aligned inpainter and selected through a guarded tournament that accepts edits only when they improve policy compliance while preserving prompt fidelity and perceptual quality. Across five jailbreak families SneakPrompt, MMA, PGJ, DACA, and RABell and UNet/flow-matching models including SD~1.5, SDXL, SD~3.5, and FLUX.1-dev. GuardPaint reduces attack success and harmful generations with minimal degradation to image quality, prompt fidelity, and benign behavior. Content warning: This paper contains examples involving nudity and violence that some readers may find disturbing, distressing, or offensive.