Training-Free Policy Violation Detection via Activation-Space Whitening in LLMs
Proposes a training-free activation-space whitening method for policy violation detection, achieving 86.0% F1 score, outperforming fine-tuned baselines by up to 9.1 points.
Key Findings
Methodology
This paper introduces a zero-shot policy violation detection framework based on activation space whitening. The core idea involves estimating the mean and covariance of in-policy hidden activations across transformer layers, then computing a linear whitening transformation W via PCA to decorrelate and standardize the features. During inference, the model’s last-layer activation vector is transformed using W, and its Euclidean norm is used as a compliance score. The approach requires only a small set of in-policy samples for statistical estimation and calibration, making it lightweight and adaptable. The detection process involves comparing the whitened activation norm against a calibrated threshold to identify out-of-policy responses. Extensive experiments on multiple LLMs (Qwen-8B, Gemma-9B, Llama-3.1-8B) and benchmarks (DynaBench, τ-bench) demonstrate the method’s robustness and superior performance over baselines.
Key Results
- The proposed whitening-based detection method achieved up to 86.0% F1 score across multiple models, outperforming fine-tuned models (e.g., DynaGuard-8B at 73.1%) and LLM-as-a-judge (GPT-4o-mini at 70.1%) by significant margins, with improvements up to 16 percentage points. The method maintained high accuracy even on models with weaker baseline performance, such as Mistral-7B, where it scored 66.8%.
- Compared to classical out-of-distribution detection techniques like Mahalanobis distance, energy scores, and KNN, the whitening method consistently yielded higher F1 scores (e.g., 82.2% vs. Mahalanobis 67.2% on Qwen-2.5-7B). It also demonstrated excellent computational efficiency, adding only 0.03-0.05 seconds per sample when using internal representations, suitable for real-time deployment.
- Ablation studies revealed that using a small number of samples (~100 per policy) for statistical estimation and selecting the optimal layer improved detection stability and performance. The method also generalized well to different policy domains, including airline trajectory data, with high AUC scores (up to 0.87), indicating strong out-of-distribution separation capability.
Significance
This work addresses a critical challenge in deploying large language models in sensitive environments: ensuring policy compliance efficiently without costly retraining. By leveraging internal activation signals, the proposed method offers a scalable, low-latency solution that can adapt to evolving policies with minimal overhead. It shifts the paradigm from output-based judgments to internal state analysis, opening new avenues for AI governance and safety. The approach’s simplicity, combined with its high accuracy, makes it highly practical for enterprise deployment, enabling continuous monitoring and compliance assurance in real time. This advancement holds promise for broad adoption in industries requiring strict adherence to legal, ethical, and organizational standards, ultimately contributing to safer and more trustworthy AI systems.
Technical Contribution
The paper introduces a novel application of activation space whitening for out-of-distribution detection tailored to policy violation identification. Unlike existing methods relying on fine-tuning or output classification, this approach directly models the distribution of in-policy hidden states, using PCA-based whitening to decorrelate features. The key technical innovation lies in estimating the mean and covariance of in-policy activations offline, then applying a computationally inexpensive linear transformation during inference to produce standardized features. The violation score is derived from the Euclidean norm of whitened activations, enabling a simple yet effective out-of-distribution detection mechanism. This method provides theoretical guarantees of distribution alignment and robustness, with minimal additional computational cost, facilitating scalable deployment across large models and dynamic policy sets.
Novelty
This study is pioneering in framing policy violation detection as an out-of-distribution problem within the activation space of large language models, utilizing a training-free whitening transformation. Unlike prior works that depend on supervised fine-tuning or output classifiers, this approach leverages the intrinsic statistical structure of internal representations, requiring only a small calibration set. The combination of PCA whitening with norm-based scoring for policy compliance detection is a novel contribution, enabling high accuracy and efficiency. It fundamentally shifts the detection paradigm from output-level judgments to internal state analysis, offering a scalable and adaptable solution that can handle complex, evolving policies without retraining.
Limitations
- The method’s performance depends on the quality and representativeness of the in-policy samples used for statistical estimation. Insufficient or biased samples may impair detection accuracy.
- In highly nuanced or ambiguous policy scenarios, the activation distribution may overlap significantly, reducing the method’s discriminative power.
- While computationally efficient, the approach still requires offline estimation and calibration, which may be challenging in rapidly changing policy environments or with extremely large models. Further research is needed to automate and streamline these processes.
Future Work
Future directions include integrating adaptive online updating of the whitening parameters to accommodate policy evolution, extending the framework to multimodal models incorporating images or audio, and exploring hierarchical or multi-layer anomaly detection strategies. Additionally, combining this activation-space approach with other interpretability techniques could enhance understanding of policy violations at the model’s internal reasoning level, fostering more transparent and controllable AI systems.
AI Executive Summary
The rapid deployment of large language models (LLMs) across various industries has revolutionized natural language processing, enabling applications from automated customer support to legal document analysis. However, as these models are integrated into sensitive domains, ensuring their adherence to organizational policies and external regulations becomes paramount. Traditional safety mechanisms, such as content filtering and prompt engineering, are often limited to coarse safety objectives like toxicity or bias detection, failing to capture the nuanced, context-dependent policies that govern real-world applications.
Existing approaches to policy compliance primarily rely on fine-tuning models or deploying external classifiers that judge the generated responses. While effective to some extent, these methods are computationally expensive, introduce latency, and lack flexibility in adapting to policy updates. Moreover, they only evaluate compliance at the output level, neglecting the rich internal representations that encode the model’s decision-making process.
Addressing these limitations, the authors propose a novel, training-free framework that leverages the internal activation states of LLMs for policy violation detection. Inspired by whitening techniques from computer vision, the method involves estimating the statistical distribution of in-policy hidden activations across transformer layers, then applying PCA-based linear transformations to decorrelate and standardize these features. During inference, the model’s last-layer activation vector is whitened using precomputed parameters, and its Euclidean norm serves as a compliance score. Responses with scores exceeding a calibrated threshold are flagged as policy violations.
This approach offers several advantages: it requires only a small set of policy-specific samples for calibration, does not necessitate model fine-tuning, and incurs minimal computational overhead—adding only milliseconds per sample. Extensive experiments on multiple open-source models, including Qwen-8B, Gemma-9B, and Llama-3.1-8B, demonstrate that the method achieves up to 86.0% F1 score, surpassing fine-tuned and judgment-based baselines by significant margins. It also outperforms classical out-of-distribution detection techniques, confirming its robustness and scalability.
The implications of this work are profound. It provides a practical tool for organizations to continuously monitor and enforce complex policies in real time, reducing legal and reputational risks. Its low-cost, adaptable nature makes it suitable for large-scale deployment, even in environments with rapidly evolving policies. Looking ahead, integrating online adaptation, multi-modal data, and interpretability features could further enhance the system’s effectiveness, paving the way for safer, more trustworthy AI systems in sensitive domains.
Deep Dive
Abstract
As organizations increasingly deploy LLMs in sensitive domains such as legal, financial, and medical settings, ensuring alignment with internal organizational policies has become a priority. Existing content moderation frameworks remain largely confined to the safety domain and lack the robustness to capture nuanced organizational policies. LLM-as-a-judge and fine-tuning approaches, though flexible, introduce significant latency and training cost. To address these limitations, we frame policy violation detection as an out-of-distribution (OOD) problem in the model's activation space. We propose a training-free method that operates directly on the LLM internal representations, leveraging prior evidence that decision-relevant information is encoded within them. Inspired by whitening techniques, we apply a linear transformation to decorrelate and standardize the model's hidden activations, and use the Euclidean norm in this transformed space as a compliance score for detecting policy violations. Our method requires only the policy text and a small number of illustrative samples, making it lightweight and easily deployable. We extensively evaluate our method across multiple LLMs and challenging policy benchmarks, achieving 86.0% F1 score while outperforming fine-tuned baselines by up to 9.1 points and LLM-as-a-judge by 16 points, with significantly lower computational cost. Code is available at: https://github.com/FujitsuResearch/LLM-policy-violation-detection
References (20)
DynaGuard: A Dynamic Guardian Model With User-Defined Policies
Monte Hoover, Vatsal Baherwani, Neel Jain et al.
DynaGuard: A Dynamic Guardrail Model With User-Defined Policies
Monte Hoover, Vatsal Baherwani, Neel Jain et al.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shun-Yu Yao, Noah Shinn, Pedram Razavi et al.
Contrastive Out-of-Distribution Detection for Pretrained Transformers
Wenxuan Zhou, Fangyu Liu, Muhao Chen
Attention is All you Need
Ashish Vaswani, Noam Shazeer, Niki Parmar et al.
Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data
Yen-Chang Hsu, Yilin Shen, Hongxia Jin et al.
Discriminative Decorrelation for Clustering and Classification
Bharath Hariharan, Jitendra Malik, D. Ramanan
DIESEL: A Lightweight Inference-Time Safety Enhancement for Language Models
Ben Ganon, Alon Zolfi, O. Hofman et al.
Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: EMNLP 2023 - Industry Track, Singapore, December 6-10, 2023
Federal Trade Commission
Earl W. Kintner, Chairman, Robert T. Secrest et al.
SOFTWARE CONSIDERATIONS IN AIRBORNE SYSTEMS AND EQUIPMENT CERTIFICATION
H. Lougee
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks
Dan Hendrycks, Kevin Gimpel
Pretrained Transformers Improve Out-of-Distribution Robustness
Dan Hendrycks, Xiaoyuan Liu, Eric Wallace et al.
ATOM: Robustifying Out-of-Distribution Detection Using Outlier Mining
Jiefeng Chen, Yixuan Li, Xi Wu et al.
Out-of-distribution Detection with Deep Nearest Neighbors
Yiyou Sun, Yifei Ming, Xiaojin Zhu et al.
Progress measures for grokking via mechanistic interpretability
Neel Nanda, Lawrence Chan, Tom Lieberum et al.
Fine-Tuning Deteriorates General Textual Out-of-Distribution Detection by Distorting Task-Agnostic Features
Sishuo Chen, Wenkai Yang, Xiaohan Bi et al.
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
Yang Liu, Yuanshun Yao, Jean-François Ton et al.
NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails
Traian Rebedea, R. Dinu, Makesh Narsimhan Sreedhar et al.
Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations
Hakan Inan, K. Upasani, Jianfeng Chi et al.
Cited By (5)
What Do Compliance Detectors Read? An Audit of Activation Probes and Guard Models
RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems
Training-free Detection of Generated Videos via Spatial-Temporal Likelihoods
Inference-Time Backdoors via Chat Templates: From LLM Supply Chains to Agentic System Compromise
LADFA: A Framework of Using Large Language Models and Retrieval-Augmented Generation for Personal Data Flow Analysis in Privacy Policies