Scalable Hierarchical Attention Transformers for Multi-Turn Jailbreak Detection in Long Conversations

The rapid development of large language models (LLMs) like GPT and BERT has revolutionized dialogue systems, enabling more natural and versatile interactions. However, these models are vulnerable to multi-turn jailbreak attacks, where adversaries gradually steer conversations toward unsafe content through tactics like escalation, role manipulation, and reframing. Early detection methods focused on turn-by-turn filtering or toxicity classifiers, which are insufficient against sophisticated multi-turn strategies. Recent research has explored dialogue-level modeling, but many approaches rely on concatenating entire conversation histories, resulting in quadratic computational costs and limited scalability. Hierarchical architectures, inspired by document classification and dialogue modeling, have shown promise in long-text tasks, but their application to safety detection remains underexplored. This paper builds on these advances, aiming to develop an efficient, scalable model capable of understanding complex dialogue dynamics to improve safety in real-world systems.

Detecting multi-turn jailbreaks poses unique challenges: attacks are often subtle, distributed across multiple turns, and involve role or topic shifts that evade simple filtering. Existing methods struggle with long conversations due to the quadratic complexity of self-attention when concatenating entire histories. The core problem is designing a model that can efficiently process long dialogues, capturing both local details and global patterns, while maintaining high accuracy and low false positives. Achieving real-time performance in large-scale deployment adds further constraints. Addressing these issues requires innovative architectures that balance computational cost with nuanced reasoning capabilities, especially under adaptive attack strategies.

This work introduces a hierarchical Transformer framework tailored for multi-turn jailbreak detection. Key innovations include: 1) a TurnEncoder based on a pre-trained multilingual model that encodes each turn independently, producing compact summaries and detailed token embeddings; 2) a ConvTransformer at the conversation level that models turn summaries with bidirectional self-attention, enhanced by cross-attention over token-level embeddings, enabling the model to zoom into critical details; 3) an attention-based pooling mechanism that weights turns according to their importance, capturing escalation and role shifts dynamically. This design effectively reduces the quadratic complexity of full-sequence attention, enabling scalable long-dialogue modeling while preserving fine-grained evidence. The architecture outperforms existing baselines in both accuracy and efficiency, representing a significant step forward in dialogue safety detection.

• �� Encode each turn independently using a pre-trained multilingual encoder (intfloat/multilingual-e5-base), generating a turn summary (CLS embedding) and token-level embeddings.
• �� Assemble turn summaries into a sequence, prepend a learnable [CONV_CLS] token, and add position and role embeddings.
• �� Model the sequence with a ConvTransformer, applying bidirectional self-attention to capture global dialogue patterns.
• �� Perform cross-attention in each layer over token-level embeddings to focus on salient words or phrases within turns.
• �� Use an attentive pooling layer to learn turn importance weights, producing a weighted dialogue representation.
• �� Final classification via a linear layer outputs safe/jailbreak probabilities, trained with binary cross-entropy loss on annotated data.
• �� During inference, encode turns independently, run the conversation module, and produce a conversation-level prediction, optionally with turn-level risk scores.

The authors trained and evaluated their model on a curated dataset of 98,626 conversations, balanced between safe and jailbreak examples. The dataset includes real-world dialogues from UltraChat, WildChat, ShareGPT, and synthetic attack scenarios covering escalation, role manipulation, and insertion attacks. Evaluation metrics include F1, precision, and recall, with comparisons against baseline models like full-sequence Transformers. Ablation studies removed components such as cross-attention to measure their impact. Hyperparameters like maximum turn length (256 tokens) and conversation length (24 turns) were optimized for efficiency. The model's robustness was tested across various attack types and dialogue lengths, demonstrating consistent high performance.

The hierarchical model achieved an F1 score of 0.9394, outperforming Claude Opus 4.7 (0.9294). It halved the false-positive rate and maintained high precision (95.90%) and recall (92.10%). Ablation experiments showed that removing cross-attention increased false positives by 2.26 percentage points, confirming its importance. The model effectively detected diverse attack strategies, including gradual escalation and sudden insertion, especially in long conversations. These results validate the architecture's ability to balance efficiency and detection accuracy, making it suitable for deployment in real-world safety systems.

This model can be integrated into large-scale dialogue platforms, such as customer service or AI assistants, to provide real-time safety monitoring. Its ability to detect subtle multi-turn attacks enhances user safety and content integrity. The architecture's efficiency allows deployment in resource-constrained environments, and its robustness supports compliance with safety standards. Future extensions could include multimodal inputs, online learning, and adaptation to emerging attack strategies, broadening its applicability across diverse AI systems.

Despite its strengths, the model relies on extensive annotated datasets, limiting generalization to unseen attack types. Handling dialogues longer than 24 turns or with high noise levels remains challenging. Computational costs for training and inference are significant, hindering deployment on edge devices. Future work should focus on model compression, unsupervised learning, and domain adaptation to address these limitations and improve scalability and robustness.