FORTIS: Benchmarking Over-Privilege in Agent Skills
FORTIS benchmark quantifies over-privilege in large models' skill selection and execution, with failure rates exceeding 62.5%, revealing systemic permission control issues.
Key Findings
Methodology
This paper introduces FORTIS, a benchmark designed to evaluate over-privilege behaviors in agent skill layers by decomposing the problem into two tasks: skill selection and skill-grounded tool execution. In Task 1, models must select the minimally sufficient skill from a large, overlapping library, ensuring no unnecessary privilege escalation. In Task 2, models are evaluated on whether they execute the assigned skill within its documented scope, avoiding expansion into broader tools or actions. The benchmark spans three domains—email, e-commerce, and filesystem—with 600 skill selection and 1543 tool execution instances. Evaluation metrics include exact match (EM), failure rate (FR), over-privilege rate (OPR), and no-action rate (NAR). Experiments with ten state-of-the-art models reveal high over-privilege behaviors, especially under realistic ambiguous conditions, with failure rates exceeding 60%. The design incorporates explicit privilege hierarchies and parameter gradients to facilitate systematic assessment of permission boundaries.
Key Results
- Across ten advanced models, the highest skill selection failure rate was 52.7% (GPT-5.4), and the highest tool execution failure rate was 62.5% (GPT-5.5), indicating frequent over-privilege behaviors. Even the best models misroute over half of the requests, primarily selecting broader skills or tools than necessary.
- In realistic scenarios with ambiguous language, failure rates soared above 75%, demonstrating models’ inability to reliably adhere to least-privilege principles in natural language understanding and permission inference.
- Models consistently favored higher-privilege tools, with over-privilege rates approaching 27% in skill selection and over 66% in tool execution, revealing systemic overreach that could pose safety risks in deployment. Failures were predominantly due to misinterpretation of natural language cues and insufficient boundary awareness.
Significance
This research uncovers fundamental safety vulnerabilities in current large language models, emphasizing that skill layer design alone cannot guarantee safe behavior. The high prevalence of over-privilege behaviors under realistic conditions highlights the urgent need for systematic permission control mechanisms. The findings challenge the assumption that larger models inherently become safer, instead revealing that without explicit boundary enforcement, models tend to overreach, risking security breaches and unintended actions. The FORTIS benchmark provides a crucial tool for evaluating and guiding future model development toward safer, more trustworthy AI systems, with implications for deployment in sensitive domains such as healthcare, finance, and autonomous systems.
Technical Contribution
The paper introduces a novel two-stage evaluation framework that explicitly models skill-based permission boundaries, integrating a hierarchical privilege structure with parameter gradients to facilitate systematic assessment. It innovatively combines natural language descriptions with explicit privilege levels, enabling models to recover operational scope solely from documentation. The benchmark’s design allows for detailed analysis of over-privilege behaviors across multiple domains and scenarios, providing a comprehensive diagnostic tool that surpasses task completion metrics. This approach advances the state-of-the-art in AI safety evaluation by focusing on permission adherence, an aspect often overlooked in existing benchmarks, and offers a foundation for developing models with built-in permission awareness.
Novelty
This work is the first to formalize skill layers as explicit privilege boundaries and evaluate models’ adherence to least-privilege principles systematically. Unlike prior benchmarks that focus solely on task success, FORTIS emphasizes permission control, integrating hierarchical privilege levels with parameter gradients and overlapping capabilities. Its two-task decomposition enables precise diagnosis of overreach at both selection and execution stages, providing a nuanced understanding of model behavior. This paradigm shift from task-centric to permission-centric evaluation marks a significant advancement in AI safety research, setting a new standard for model assessment.
Limitations
- The benchmark relies on predefined privilege hierarchies and static documentation, which may not fully capture dynamic permission adjustments or context-dependent behaviors in real-world systems.
- The evaluation primarily focuses on static, natural language-based permissions, lacking integration with multi-modal inputs or real-time permission management mechanisms.
- Models exhibit severe failures in boundary-sensitive scenarios, indicating that current training paradigms do not sufficiently instill permission awareness, necessitating further research into training objectives and architectures.
Future Work
Future research will explore adaptive permission management strategies, incorporating reinforcement learning and human-in-the-loop supervision to improve models' understanding of dynamic boundaries. Expanding the benchmark to include multi-modal inputs and real-time permission adjustments will enhance its applicability. Additionally, integrating explainability modules could help interpret permission decisions, fostering trust and transparency. Long-term, the goal is to develop inherently safe models with built-in permission awareness, capable of operating reliably across diverse, complex environments while adhering to strict safety standards.
AI Executive Summary
The rapid deployment of large language models (LLMs) in diverse real-world applications has raised critical concerns about their safety and controllability. While these models demonstrate remarkable capabilities in tasks ranging from natural language understanding to decision-making, their behavior at the permission boundary remains largely unregulated. Traditional evaluation metrics focus on task success rates, neglecting whether models adhere to the principle of least privilege—a fundamental concept in security that mandates only the necessary permissions for a given task.
This paper introduces FORTIS, a comprehensive benchmark designed to quantify over-privilege behaviors in agent skill layers. The core idea is to treat skills as explicit permission boundaries, which models should respect during both skill selection and execution. The benchmark decomposes the problem into two tasks: first, whether the model can identify the minimally sufficient skill from a large, overlapping library; second, whether it faithfully executes that skill without expanding into broader tools or actions. By organizing skills and tools into hierarchical privilege levels, and allowing overlaps, the benchmark simulates realistic ambiguity and decision-making scenarios.
Experimental results across ten state-of-the-art models reveal a troubling pattern: even the most advanced models fail to reliably exercise restraint. The failure rates in skill selection reach over 50%, and in skill-grounded tool execution, over 62%, with failure rates climbing above 75% in ambiguous, real-world-like scenarios. These findings highlight a systemic issue—models tend to prefer higher-privilege capabilities, often without explicit adversarial prompts, simply due to natural language vagueness and boundary proximity.
The significance of this work lies in its paradigm shift. Instead of solely focusing on task success, it emphasizes permission adherence as a core safety metric. The innovative two-stage evaluation framework and hierarchical privilege design provide a new lens for understanding and improving model safety. It challenges the community to rethink model training and architecture, integrating explicit permission control mechanisms.
Looking ahead, future research should focus on dynamic permission management, multi-modal boundary detection, and explainability. Developing models that inherently understand and respect operational boundaries will be crucial for deploying AI in sensitive domains such as healthcare, finance, and autonomous systems. Overall, FORTIS offers a vital step toward safer, more trustworthy AI, aligning technical progress with security imperatives in an increasingly AI-driven world.
Deep Dive
Abstract
Large language model agents increasingly operate through an intermediate skill layer that mediates between user intent and concrete task execution. This layer is widely treated as an organizational abstraction, but we argue it is also a privilege boundary that current models routinely exceed. We present \textbf{FORTIS}, a benchmark that evaluates over-privilege in agent skills across two stages: whether a model selects the minimally sufficient skill from a large overlapping library, and whether it executes that skill without expanding into broader tools or actions than the skill permits. Across ten frontier models and three domains, we find that over-privileged behavior is the norm rather than the exception. Models consistently reach for higher-privilege skills and tools than the task requires, failing at both stages at rates that remain high even for the strongest available models. Failure is especially severe under the ordinary conditions of real user interaction: incomplete specification, convenience framing, and proximity to skill boundaries. None of these requires adversarial construction. The results indicate that the skill layer, far from containing agent behavior, is itself a primary source of privilege escalation in current systems.
References (20)
Gorilla: Large Language Model Connected with Massive APIs
Shishir G. Patil, Tianjun Zhang, Xin Wang et al.
InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents
Qiusi Zhan, Zhixiang Liang, Zifan Ying et al.
R-Judge: Benchmarking Safety Risk Awareness for LLM Agents
Tongxin Yuan, Zhiwei He, Lingzhong Dong et al.
Testing Language Model Agents Safely in the Wild
Silen Naihin, David Atkinson, Marc Green et al.
Biased-Predicate Annotation Identification via Unbiased Visual Predicate Representation
Li Li, Chenwei Wang, Youxuan Qin et al.
Toward Complex-query Referring Image Segmentation: A Novel Benchmark
Wei Ji, Li Li, Hao Fei et al.
A survey on large language model based autonomous agents
Lei Wang, Chengbang Ma, Xueyang Feng et al.
ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs
Yujia Qin, Shi Liang, Yining Ye et al.
Panoptic Scene Graph Generation with Semantics-prototype Learning
Li Li, Wei Ji, Yiming Wu et al.
WebArena: A Realistic Web Environment for Building Autonomous Agents
Shuyan Zhou, Frank F. Xu, Hao Zhu et al.
Voyager: An Open-Ended Embodied Agent with Large Language Models
Guanzhi Wang, Yuqi Xie, Yunfan Jiang et al.
τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains
Shunyu Yao, Noah Shinn, Pedram Razavi et al.
Transfer Visual Prompt Generator across LLMs
Ao Zhang, Hao Fei, Yuan Yao et al.
Generative AI
S. Feuerriegel, Jochen Hartmann, Christian Janiesch et al.
API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs
Minghao Li, Feifan Song, Yu Bowen et al.
Toolformer: Language Models Can Teach Themselves to Use Tools
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì et al.
ReAct: Synergizing Reasoning and Acting in Language Models
Shunyu Yao, Jeffrey Zhao, Dian Yu et al.
On the Importance of Building High-quality Training Datasets for Neural Code Search
Zhensu Sun, Li Li, Y. Liu et al.
SRDiff: A Cross-Modal Diffusion Model for Satellite-to-Radar Translation in Precipitation Nowcasting
Youxuan Qin, Jinming Cao, Tingyun Wang et al.
Generalized Video Moment Retrieval
Youxuan Qin, Qilong Wu, Yicong Li et al.
Cited By (7)
Agent Safety Is Action Alignment
Task-Conditioned Least-Privilege Learning for Executable Terminal and MCP Agents
OmniScientist: An Omni-Modal Omni-Discipline AI Scientist
From Profiling to Synthesis: Benchmarking Implicit Behavioral Alignment in Personalized LLM Agents
JigShape: Evaluating Visual-Geometric Reasoning in VLMs through Jigsaw Puzzles
RaMem: Contextual Reinstatement for Long-term Agentic Memory
Auditable Agents