Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Models

The evolution of LLMs such as GPT-3, GPT-4, and PaLM has transformed AI applications across industries. Early efforts focused on bias mitigation, privacy, and interpretability, with techniques like Reinforcement Learning from Human Feedback (RLHF) improving alignment. Nonetheless, the open nature of these models exposes them to malicious exploitation, notably through jailbreak prompts—natural language instructions designed to bypass safety constraints. Online communities have proliferated with various prompts, revealing vulnerabilities. Existing defenses are largely black-box, relying on post-hoc filtering, which is insufficient against adaptive attacks. The need for systematic understanding and automated tools to analyze and counteract jailbreak strategies has become urgent.

Despite advancements, large language models remain susceptible to prompts that can induce harmful outputs, undermining safety guarantees. The core challenge lies in the diversity and subtlety of effective jailbreak prompts, which can be crafted by users with minimal expertise. Current detection methods lack robustness against evolving strategies, and manual analysis is labor-intensive. Automating the identification and generation of effective jailbreak prompts is essential for proactive defense. Moreover, understanding the underlying mechanisms that make certain prompts successful remains an open question, critical for designing resilient models. The problem is compounded by the rapid pace of community-driven prompt sharing and innovation, which outstrips traditional security measures.

This work introduces a comprehensive taxonomy of jailbreak prompts, categorizing 448 prompts into five classes and ten patterns, providing a structured understanding of attack strategies. It proposes two novel metrics—probability of bypass and response detail—to quantitatively evaluate effectiveness. The core innovation is the development of an iterative prompt mutation framework, inspired by fuzz testing, that automatically refines prompts to maximize success rates. This system leverages human insights and model feedback, enabling scalable and adaptive attack simulations. Additionally, the open-sourcing of datasets and tools fosters community engagement, bridging the gap between theoretical analysis and practical defense. These contributions collectively enhance the understanding and mitigation of jailbreak vulnerabilities.

• �� Data collection: Scraped 448 prompts from online forums, repositories, and social media, manually verified duplicates. • Categorization: Applied thematic analysis to classify prompts into Disguised Intent, Role Play, Structured Response, Virtual AI Simulation, and Hybrid Strategies. • Effectiveness metrics: Designed two measures—success probability and response detail—assessed via human annotation across multiple models. • Automated system: Developed an iterative prompt mutation pipeline, inspired by fuzz testing, where • prompts are modified using predefined patterns, • responses are evaluated against metrics, • successful prompts are retained and further mutated. • User study: Recruited 92 participants with diverse backgrounds to analyze manual prompt creation strategies. • Validation: Tested the system on 766 previously failed prompts, achieving high success rates, and compared results across GPT-3.5, GPT-4, and PaLM-2.

The dataset comprised 448 prompts and 161 malicious queries aligned with OpenAI policies, covering scenarios like harmful instructions, hate speech, misinformation, and malware. Human annotators evaluated model responses, assigning success scores based on the ability to elicit prohibited content. The automated system iteratively mutated prompts, with each cycle tested on multiple models. Results showed that the system transformed 729 out of 766 prompts into successful jailbreaks, with success rates exceeding 95%. Cross-model comparisons indicated GPT-4 was most vulnerable, with an 85% success rate using the identified strategies. Ablation studies confirmed the effectiveness of prompt mutation patterns, and the metrics correlated well with human judgments, validating the evaluation framework.

The classification revealed that certain prompt patterns, especially Role Play and Virtual AI Simulation, are most effective in bypassing defenses. The automated system demonstrated rapid scalability, transforming nearly all failed prompts into successful ones within a few iterations. Human participants, regardless of expertise, could generate complex prompts, emphasizing the accessibility of jailbreak strategies. The success metrics correlated strongly with actual model outputs, confirming their validity. These findings underscore the urgent need for more robust defenses and highlight the potential of automated tools in security assessments.

The insights enable development of advanced detection algorithms, incorporating pattern recognition and anomaly detection. Automated prompt generation tools can be used to simulate attacks, stress-test models, and improve safety measures. Industry applications include secure deployment of chatbots, content moderation, and privacy protection. Policymakers can leverage these findings to formulate standards for model safety and transparency, ensuring responsible AI usage. Researchers can further explore adaptive defenses, integrating real-time monitoring and dynamic prompt filtering, to stay ahead of evolving jailbreak techniques.

The study focuses on text-based models, with limited exploration of multimodal systems. Automated system performance may vary with more complex prompts or unseen patterns. Dataset bias towards community-shared prompts could limit generalization. Defensive measures need to evolve to counter adaptive, multi-stage attacks. Future work should include broader model architectures, real-world deployment scenarios, and continuous learning mechanisms to enhance robustness.