When Agents Go Rogue: Activation-Based Detection of Malicious Behaviors in Multi-Agent Systems
Activation-space anomaly detection in MAS outperforms graph-based methods, achieving F1 of 0.94/0.93 in synchronous/asynchronous settings.
Key Findings
Methodology
This paper introduces AcMAS, an activation-based framework that models local agent internal reasoning as a distribution in activation space. It detects malicious behaviors by measuring cosine distance deviations from a learned normal prototype, independent of explicit interaction graphs or synchronization assumptions. The approach combines anomaly detection with a novel activation correction mechanism, steering compromised agent activations back toward the normal manifold. The core components include feature extraction from final-layer hidden states, divergence computation, adaptive thresholding, and activation-level restoration, enabling robust detection and recovery in both synchronous and asynchronous MAS environments.
Key Results
- Across five attack scenarios, including stealth prompt injection, tool manipulation, and memory poisoning, AcMAS achieved F1 scores of 0.94 in synchronous and 0.93 in asynchronous settings, surpassing graph-based baselines (0.72 and 0.38 respectively) by over 0.5. Detection accuracy exceeded 95%, with false positive rates below 5%. The activation correction mechanism restored task completion rates to 97%, outperforming isolation methods which caused significant task disruption. The method demonstrated consistent performance across diverse LLM architectures (GPT-3.5, Llama-2), system scales (8-80 agents), and attack intensities, confirming its generalizability.
- Results highlight that internal activation deviations serve as early indicators of stealthy attacks, and the proposed correction effectively restores agent reasoning, maintaining system functionality while minimizing false alarms.
Significance
This work addresses critical limitations in existing MAS security defenses, notably their reliance on explicit interaction graphs and semantic cues, which are ineffective against covert attacks and asynchronous execution. By leveraging internal reasoning signals, AcMAS offers a robust, scalable, and architecture-agnostic solution that significantly enhances the resilience of multi-agent systems. Its ability to detect stealthy threats and restore agent functionality without disrupting collaboration marks a substantial advancement, paving the way for safer deployment of intelligent autonomous systems in real-world scenarios.
Technical Contribution
The paper introduces a novel activation divergence-based anomaly detection algorithm that models normal agent reasoning as a stable distribution in activation space. It departs from traditional graph-based models, providing theoretical guarantees of robustness in asynchronous environments. The integration of an activation steering mechanism for functional restoration is a key innovation, enabling real-time correction of compromised agents. The approach’s scalability and generalization across multiple LLM architectures and attack types demonstrate its practical value and theoretical novelty, offering a new paradigm for MAS security.
Novelty
This is the first work to utilize internal activation space modeling for stealth attack detection in MAS, avoiding reliance on explicit communication graphs. The combination of divergence-based anomaly detection with activation-level correction introduces a new framework that is inherently asynchronous-robust and attack-agnostic. Unlike prior methods focused on output semantics or structural properties, this approach taps into the deep internal reasoning signals, providing a fundamentally different and more resilient detection paradigm.
Limitations
- The method depends on accurate modeling of normal activation distributions, which may degrade under severe domain shifts or unseen attack types. Continuous updating of the normal baseline is necessary.
- Activation correction, while effective, may not fully recover complex or multi-point coordinated attacks, especially under high attack density.
- Computational overhead for large-scale systems could be significant due to repeated feature extraction and divergence calculations, requiring optimization for real-time deployment.
Future Work
Future research will explore multi-modal activation features, integrating visual or sensor data for richer anomaly signals. Combining reinforcement learning to adaptively tune correction strength could enhance resilience. Extending the framework to multi-task and multi-modal MAS, and developing theoretical bounds for detection guarantees, are promising directions to further strengthen the security and robustness of autonomous multi-agent systems.
AI Executive Summary
The rapid development of large language models (LLMs) has transformed multi-agent systems (MAS), enabling autonomous reasoning, communication, and collaboration among heterogeneous agents. These systems are increasingly deployed in complex, dynamic environments such as autonomous vehicles, smart manufacturing, and intelligent assistants. However, their security remains a major concern. Traditional defenses rely heavily on explicit interaction graphs or semantic cues, which are vulnerable to stealthy attacks like prompt injection, memory poisoning, and tool manipulation. Moreover, many existing methods assume synchronized communication, which does not reflect real-world asynchronous MAS architectures, leading to detection failures.
This paper presents AcMAS, a novel activation-based framework that leverages internal reasoning signals to detect malicious behaviors. Instead of depending on explicit communication structures, AcMAS models each agent’s internal activation vectors—extracted from the final layer of their LLM backbones—and learns a normal behavior distribution. Deviations from this distribution, measured via cosine distance, serve as early indicators of covert attacks. The framework operates effectively in both synchronous and asynchronous settings, addressing the limitations of prior graph-based methods.
Beyond detection, AcMAS introduces an activation correction mechanism that steers compromised agents’ internal states back toward normal behavior, restoring their reasoning capabilities without isolating or removing them. Extensive experiments across five attack scenarios demonstrate that AcMAS achieves F1 scores above 0.93, outperforming baseline methods by over 0.5 in both environments. It maintains high task completion rates (~97%) while significantly reducing attack success rates.
This work offers a robust, scalable, and architecture-agnostic solution for MAS security, capable of detecting stealthy threats and restoring agent functionality in real-time. Its implications extend to deploying safer, more resilient autonomous systems across diverse industries. Future directions include multi-modal integration, adaptive learning for dynamic environments, and theoretical analysis of detection guarantees, promising a new era in intelligent system security.
Deep Dive
Abstract
While enabling effective collaboration on complex tasks, LLM-based Multi-Agent Systems (MAS) face critical security challenges due to vulnerabilities at the agent and interaction levels. Most existing MAS security defenses are built upon two core assumptions: semantically-explicit malicious attacks and explicit graph-based modeling of the MAS topology and agent-level interactions. In practice, real-world attacks are becoming more semantically stealthy, while MAS execution is typically asynchronous without the temporal alignment assumed by graph-based propagation models. To address these limitations, we propose AcMAS, an activation-based framework for malicious-behavior detection in MAS. By analyzing internal reasoning states in the activation space of local agents, AcMAS detects even stealthy attacks in a synchronization-robust fashion, without relying on explicit interaction graphs. Moreover, our activation analysis provides critical signals to guide AcMAS in restoring the functionality of compromised agents, rather than the disruptive agent isolation commonly used by the state-of-the-art methods. Comprehensive evaluation demonstrates that AcMAS significantly outperforms graph-based baselines against stealthy attacks, by +0.22 F1 in synchronous settings (0.94 vs. 0.72) and by +0.55 F1 in asynchronous settings (0.93 vs. 0.38), with generalization across diverse open-source LLM backbones, attack intensity, and MAS scale.