How Fragile Is Safety Alignment at Frontier Scale? A Single-Direction Attack on a 320B MoE
Single-direction attack on GLM-5.3-Flash reveals fragility in safety alignment.
Key Findings
Methodology
The study employs directional ablation to remove the 'refusal direction' from a language model, requiring no gradient training, only a few hundred contrastive prompts. This method is applied to GLM-5.3-Flash, a 320B parameter model with 288 experts, using a four-stream hyper-connection residual and block-FP8 quantization.
Key Results
- On GLM-5.3-Flash, editing attention, dense, and expert modules individually removes 0.039, 0.016, and 0.148 of refusal, respectively; editing all three together removes 0.776, showing 74% of the effect exists only under joint intervention.
- The conventional method removes only 0.066 of refusal via module-name matching, failing to address MoE architecture effectively.
- The method achieves 41-89 percentage-point reductions across seven harmful benchmarks with no detected change in capability.
Significance
This study highlights the fragility of safety alignment at frontier scale, particularly in mixture-of-experts models. It demonstrates the limitations of traditional directional ablation methods in handling complex architectures, emphasizing the need to reassess safety alignment strategies in multi-stream residual and quantized environments.
Technical Contribution
The study shows the applicability of directional ablation in multi-stream residuals and quantized weights, highlighting that editing residual-writing weights is more thorough than hooking the layer boundary. It also reveals that refusal signals in MoE models are no longer confined to a single direction.
Novelty
This is the first validation of directional ablation on a 320B parameter mixture-of-experts model, revealing the diversity and distribution of refusal signals in complex architectures, challenging the traditional single-direction hypothesis.
Limitations
- The method may not completely remove all refusal signals in multi-stream residuals, especially those concentrated in specific content categories.
- Editing quantized weights may introduce additional engineering complexity.
Future Work
Future research could explore safety alignment strategies under different architectures and quantization schemes, focusing on enhancing adversarial robustness without compromising model capability.
AI Executive Summary
The study explores the fragility of safety alignment in language models at frontier scale, particularly in mixture-of-experts (MoE) architectures. Traditional directional ablation methods remove the 'refusal direction' from models, but their effectiveness in complex architectures remains unverified.
The study applies directional ablation to the GLM-5.3-Flash model, finding that individual edits to attention, dense, and expert modules have limited effect, while joint editing significantly enhances results. This indicates that in complex architectures, refusal signals are no longer confined to a single direction.
Experimental results show significant reductions in refusal rates across seven harmful benchmarks, with no noticeable change in model capability. The study emphasizes the need to reassess safety alignment strategies in multi-stream residual and quantized environments, providing new perspectives for future safety alignment research.
Deep Analysis
Background
As language models scale, safety alignment becomes crucial. Traditional directional ablation methods have been validated on smaller dense models but not on mixture-of-experts models. GLM-5.3-Flash features complex architectures, including multi-stream residuals and quantized weights, posing new challenges.
Core Problem
The core problem is effectively removing refusal signals in complex mixture-of-experts architectures. Traditional methods assume refusal signals are confined to a single direction, which may not hold in multi-stream residuals and quantized weights.
Innovation
The study first validates directional ablation on a 320B parameter mixture-of-experts model, revealing the diversity and distribution of refusal signals in complex architectures. Joint editing of multiple modules significantly enhances refusal signal removal.
Methodology
- �� Use directional ablation to remove the 'refusal direction' from language models.
- �� Apply to GLM-5.3-Flash, a model with 320B parameters and 288 experts.
- �� Evaluate the effects of individual and joint editing of attention, dense, and expert modules.
Experiments
Experiments conducted on GLM-5.3-Flash use seven harmful benchmarks to evaluate the method's effectiveness. By comparing the effects of individual and joint editing of different modules, the applicability of directional ablation in complex architectures is validated.
Results
Results show that individual edits to attention, dense, and expert modules have limited effect, while joint editing significantly enhances results. The method achieves 41-89 percentage-point reductions in refusal rates across seven harmful benchmarks.
Applications
The findings can improve safety alignment strategies for language models, particularly in handling complex architectures and quantized weights. It provides new insights for developing more robust safety alignment methods.
Limitations & Outlook
While effective in complex architectures, the method may not completely remove all refusal signals in multi-stream residuals, especially those concentrated in specific content categories. Additionally, editing quantized weights may introduce additional engineering complexity.
Plain Language Accessible to non-experts
Imagine a massive library with many shelves, each holding different books. You want to ensure certain books, like those with inappropriate content, aren't borrowed. Traditional methods label these books, but someone finds a way to hide them by rearranging the shelves. This study explores how to rearrange the shelves to ensure inappropriate content isn't borrowed without affecting other books.
ELI14 Explained like you're 14
Imagine you're playing a game where your task is to stop bad guys from entering a castle. You have a button to close the gate, but some bad guys are clever and always find a way in. This study is like finding a new method to keep these clever bad guys out, no matter what, without stopping the good guys from entering. Cool, right?
Glossary
Directional Ablation
A method to remove specific behaviors from a model by projecting out certain directions, requiring no gradient training.
Used to remove the 'refusal direction' in language models.
Mixture of Experts Model
An architecture that scales parameter count through conditional computation using multiple expert modules.
The architecture used by GLM-5.3-Flash.
Multi-Stream Residual
A design that splits the residual into multiple parallel streams, enhancing training stability.
The residual design of GLM-5.3-Flash.
Quantized Weights
Weights stored in a low-precision format to reduce storage and computation costs.
The weight storage format of GLM-5.3-Flash.
Refusal Direction
The linear direction in a model that leads to refusal of certain requests.
The target for removal by directional ablation.
Open Questions Unanswered questions from this research
- 1 How to enhance refusal signal removal without affecting model capability.
- 2 The distribution characteristics of refusal signals under different architectures and quantization schemes.
Applications
Immediate Applications
Safety Alignment Optimization
Improve safety alignment strategies for language models, especially when handling complex architectures.
Long-term Vision
Robustness Enhancement
Develop more robust safety alignment methods applicable to various model architectures and quantization schemes.
Abstract
Directional ablation removes an aligned language model's ability to refuse by projecting a single "refusal direction" out of the weights that write the residual stream. It needs no gradient-based training and no optimization, only a few hundred contrastive prompts, which makes it the canonical white-box attack on open-weight alignment. However, it has been established only on dense models up to roughly 70B parameters. We study whether it survives the shift to frontier mixture-of-experts (MoE) models whose residual streams are no longer a single tensor and whose weights ship quantized. We apply it to GLM-5.3-Flash (320B parameters, 288 routed experts, a four-wide hyper-connection residual, block-FP8). The attack survives the architecture, but what it reaches is no longer where a reader of the original recipe would look for it. Editing the attention, dense and routed-expert writers on their own removes 0.039, 0.016 and 0.148 of refusal respectively; editing all three together removes 0.776. As a result, 74% of the effect exists only under the joint intervention. The part the conventional recipe reaches by module-name matching accounts for 0.066 of that 0.776, which is why it fails silently on an MoE. The effect does not follow from removing just any direction: ablating a random direction orthogonal to it leaves refusal unchanged. A category-concentrated residue survives every edit we tried: subspaces fitted on violence, sexual content and hate leave measurable refusal at every rank from 1 to 12. We report the method, the 41-89 percentage-point reductions it achieves across seven harmful benchmarks with no detected change in capability, and the boundary where it stops.