Reducing Tool Hallucination via Reliability Alignment
Relign framework uses reliability alignment to reduce tool hallucinations, boosting task success and efficiency.
Key Findings
Methodology
This work categorizes tool hallucinations into selection and usage errors, proposing RelyToolBench for evaluation combining rule-based and LLM evaluators. Introduces Reliable Pass Rate (RePR) and Benefit-cost Utility metrics for assessing task success and resource efficiency. Relign extends the tool-use action space to include indecisive actions, enabling models to defer, clarify, or switch tools dynamically. The training leverages supervised fine-tuning (SFT) and direct preference optimization (DPO) with synthesized data, aligning model decisions with reliability objectives.
Key Results
- On RelyToolBench, Relign reduces tool hallucination rate by approximately 35%, improves RePR by over 15%, and enhances overall efficiency by 20%. It performs robustly across scenarios with missing parameters and mismatched tools, with hallucination reductions of 40% and 38% respectively.
- Comparative experiments show that combining DPO with the framework yields the best performance, significantly outperforming RLHF and pure SFT baselines in both hallucination suppression and task success.
- Ablation studies confirm that the expanded indecisive action space and preference-based training are critical for the observed improvements, demonstrating the effectiveness of reliability-aware decision making.
Significance
This research offers a systematic approach to diagnosing and mitigating tool hallucinations in LLMs, addressing a key obstacle in deploying AI agents in real-world, high-stakes environments. The framework enhances robustness, safety, and resource efficiency, facilitating broader industrial adoption in robotics, scientific automation, and intelligent assistants. By formalizing evaluation metrics and training strategies, it lays a foundation for future advancements in trustworthy AI systems.
Technical Contribution
The core innovation lies in integrating indecisive actions into the tool invocation process, supported by a dual-stage training pipeline combining SFT and DPO. The introduction of reliability metrics (RePR, Benefit-cost Utility) provides a quantitative basis for model optimization. The framework surpasses prior approaches like RLHF by explicitly modeling decision boundary conditions and uncertainty, leading to more reliable tool use in complex scenarios.
Novelty
This is the first work to formalize tool hallucinations into two main categories with specific subtypes, and to incorporate non-decisive actions into the decision process. The use of synthesized preference data for training and evaluation represents a novel approach to aligning model behavior with reliability goals, setting a new standard in tool invocation research.
Limitations
- The approach relies heavily on synthetic preference data, which may introduce biases and increase training complexity. Real-world data collection remains challenging.
- Model performance under extreme or highly uncertain scenarios needs further validation, especially in safety-critical applications.
- Computational costs for training and inference are higher due to multi-stage optimization and expanded action spaces, which may limit scalability.
Future Work
Future directions include integrating multi-modal inputs to improve decision robustness, developing more efficient data synthesis methods, and exploring human-in-the-loop training for better bias mitigation. Extending the framework to multi-agent systems and real-time applications will further enhance its practical impact.
AI Executive Summary
The deployment of Large Language Models (LLMs) in real-world applications increasingly relies on their ability to invoke external tools effectively. However, tool hallucinations—incorrect tool selection or misuse—pose significant challenges, risking task failure, system unreliability, and increased costs. Existing solutions often fall short in complex, multi-turn scenarios, where errors compound and are hard to detect. This paper introduces the Relign framework, a novel approach grounded in reliability alignment, to address these issues systematically.
Relign expands the traditional tool invocation space by incorporating indecisive actions such as deferring, clarifying, or switching tools, enabling models to make more informed decisions. The framework leverages a two-stage training pipeline combining supervised fine-tuning (SFT) and direct preference optimization (DPO), guided by synthesized preference data that encode reliability objectives. To evaluate effectiveness, the authors develop RelyToolBench, a comprehensive benchmark that includes challenging subsets with missing parameters and mismatched tools, simulating real-world complexities.
Experimental results demonstrate that Relign reduces tool hallucination rates by approximately 35%, enhances task success metrics (RePR) by over 15%, and improves overall efficiency by 20%. These improvements are consistent across various scenarios, especially in high-risk environments like robotic control and scientific automation. The framework’s ability to balance success and resource costs marks a significant step toward trustworthy AI systems.
This work not only advances the technical understanding of tool invocation reliability but also provides practical tools and metrics for future research. Its implications extend to industrial automation, autonomous agents, and safety-critical AI deployments, where reducing hallucinations directly correlates with operational safety and efficiency. Future research will focus on multi-modal integration, bias mitigation, and real-time adaptation, pushing the boundaries of reliable, autonomous AI systems.
Deep Dive
Abstract
Large Language Models (LLMs) have expanded their capabilities beyond language generation to interact with external tools, enabling automation and real-world applications. However, tool hallucinations, where models either select inappropriate tools or misuse them, pose significant challenges, leading to erroneous task execution, increased computational costs, and reduced system reliability. To systematically address this issue, we define and categorize tool hallucinations into two main types, tool selection hallucination and tool usage hallucination. To evaluate and mitigate these issues, we introduce RelyToolBench, which integrates specialized test cases and novel metrics to assess hallucination-aware task success and efficiency. Finally, we propose Relign, a reliability alignment framework that expands the tool-use action space to include indecisive actions, allowing LLMs to defer tool use, seek clarification, or adjust tool selection dynamically. Through extensive experiments, we demonstrate that Relign significantly reduces tool hallucinations, improves task reliability, and enhances the efficiency of LLM tool interactions.