VPA-Guard: Defending and Benchmarking Image-to-Video Generation Against Visual Prompt Attacks
VPA-Guard combines retrieval-augmented reasoning and self-evolution to reduce visual prompt attack success rates by 44.2%, enhancing video generation safety.
Key Findings
Methodology
This study constructs VVA-Bench, categorizing visual prompt attacks into single-image and two-frame types, covering mechanisms like motion, emoji, pose, sketch, and camera control. It employs DINOv2 visual encoders and Sentence-BERT text encoders to fuse features, retrieving similar attack samples from a vector database. Few-shot reasoning prompts guide models to recognize malicious intents. The VPA-Guard framework integrates retrieval with a self-evolving module that updates attack knowledge dynamically through post-generation reflection. Experimental evaluation on models like Wan 2.7 and Veo 3.1 shows attack success rates of 100% and 74.8%, respectively, which are reduced by 44.2% with the defense.
Key Results
- Without defense, Wan 2.7's attack success rate hits 100%, with harmfulness scores up to 7.30. Introducing VPA-Guard reduces ASR by 44.2% and HS by 73.4%, significantly improving safety. The method performs well across attack types, especially motion and emoji-based prompts. Benign sample rejection rate remains low at 12.8%, ensuring user experience isn't compromised.
- The automated multimodal large language model (Qwen 3.5-VL-35B) judge correlates highly with human experts, validating the evaluation process. Temporal harmfulness remains challenging but acceptable, with correlations around 0.55.
- The framework's adaptability allows continuous learning of new attack paradigms, maintaining robustness against evolving threats. It balances safety and controllability, making it suitable for real-world deployment.
Significance
This work addresses a critical gap in multimodal safety by systematically evaluating and defending against visual prompt attacks in video generation. It introduces a scalable benchmark and a robust defense framework, pushing the field toward safer AI content creation. The approach enhances trustworthiness in AI-generated videos, crucial for social media, entertainment, and safety-critical applications. It also demonstrates how retrieval and self-reflection mechanisms can be combined to tackle complex, implicit threats, setting a foundation for future research and industry standards.
Technical Contribution
The key innovation lies in integrating a retrieval-augmented few-shot reasoning system with a self-evolving attack knowledge base, enabling dynamic detection of malicious visual cues. The multimodal feature fusion enhances interpretability and robustness. The multi-stage evaluation protocol, combining attack success rate and harmfulness score, provides a comprehensive safety assessment. Compared to prior methods, VPA-Guard offers superior detection of implicit and evolving threats, balancing safety and controllability effectively.
Novelty
This is the first comprehensive framework to classify, benchmark, and defend against visual prompt attacks in video generation. It innovatively combines retrieval-based attack pattern matching with self-reflection to adapt to new threats, surpassing traditional static filters and fine-tuning approaches. The establishment of VVA-Bench as a systematic benchmark is also pioneering, enabling reproducible evaluation of emerging attack vectors.
Limitations
- Despite strong performance, the system may still miss highly novel or complex attack patterns, especially those outside the current attack sample library.
- Self-evolving mechanisms depend on post-attack reflection, which could introduce false positives or negatives if misinterpreted.
- Real-time deployment requires further optimization to balance detection speed and accuracy, especially in high-throughput scenarios.
Future Work
Future research will focus on enhancing the self-evolving module with reinforcement learning, expanding attack sample diversity, and integrating more sophisticated temporal reasoning. Developing more efficient retrieval and reflection strategies will improve real-time performance. Additionally, extending the framework to other multimodal tasks, such as text-to-video or audio-visual synthesis, will broaden its applicability. Establishing industry standards for visual prompt safety is also a promising direction.
AI Executive Summary
In recent years, image-to-video (I2V) generation has experienced rapid advances, enabling the transformation of static images into dynamic videos with fine-grained control. Users can manipulate video content through visual cues like arrows, emojis, and sketches, making the process highly interactive and customizable. However, this increased controllability introduces significant safety risks. Malicious actors can embed subtle visual prompts that models interpret as executable instructions, leading to the generation of harmful or unsafe videos. Existing safety benchmarks primarily focus on text-based or content-only filters, leaving a critical gap in evaluating visual prompt vulnerabilities.
To address this, the authors developed VVA-Bench, a comprehensive benchmark that systematically categorizes and evaluates visual prompt attacks. The benchmark includes diverse attack mechanisms such as movement cues, emoji instructions, pose controls, and camera directives, covering both single-image and two-frame attack scenarios. It contains 452 unsafe attack samples and 45 safe controls, enabling detailed assessment of model vulnerabilities across multiple safety categories.
Experimental results reveal that state-of-the-art models like Wan 2.7 and Veo 3.1 are highly susceptible, with attack success rates reaching 100% and 74.8%, respectively. These attacks often produce videos with severe safety violations, reflected in high harmfulness scores. To mitigate these threats, the authors propose VPA-Guard, a novel defense framework combining retrieval-augmented reasoning with a self-evolving mechanism. This system retrieves similar attack patterns from a curated database, uses few-shot prompts to guide models toward safe outputs, and dynamically updates its knowledge base based on post-generation reflections.
Evaluation demonstrates that VPA-Guard reduces attack success rates by 44.2% and harmfulness scores by 73.4%, while maintaining legitimate editing capabilities. The approach's scalability and robustness are validated through automated assessments aligned with human judgments. Overall, this work advances the field by providing a rigorous benchmark and an effective defense strategy, fostering safer multimodal content generation. Future directions include enhancing real-time detection, expanding attack diversity, and applying these principles to broader multimodal tasks, ultimately promoting responsible AI deployment in society.
Deep Dive
Abstract
Recent advancements in Image-to-Video (I2V) generation have transformed input images from simple appearance references into interactive control interfaces where visual cues such as arrows, sketches, and emojis orchestrate complex video dynamics with unprecedented controllability. However, these seemingly innocuous static cues can be interpreted by models as executable temporal instructions, unfolding into harmful actions in the generated videos. Despite the severity of this threat, existing safety benchmarks remain predominantly focused on text-based and content-only image-based jailbreaks, leaving implicit visual prompt attacks insufficiently explored. To bridge this gap, we present VVA-Bench, the first systematic benchmark for evaluating video generation safety under categorized vision-centric prompt attacks. Extensive experiments on VVA-Bench demonstrate that state-of-the-art models are highly susceptible to such attacks, with Attack Success Rates (ASR) reaching 100.0\% on Wan 2.7 and 74.8\% on Veo 3.1. To mitigate these risks, we propose VPA-Guard, a retrieval-augmented and self-evolving defense framework. By leveraging few-shot reasoning to identify latent malicious intents, our method reduces the attack ASR by 44.2\% and the harmfulness score by 73.4\% on average, while maintaining the model's utility for legitimate user edits. Our work provides both a rigorous benchmark and an effective defense strategy to advance safe and socially responsible multimodal generation.