HalluClear: Diagnosing, Evaluating and Mitigating Hallucinations in GUI Agents
HalluClear employs classification, three-stage evaluation, and closed-loop reasoning to reduce GUI hallucinations effectively.
Key Findings
Methodology
This work introduces a GUI-specific hallucination taxonomy derived from failure analysis, combined with a three-stage automated evaluation workflow that employs expert-annotated benchmarks and ensemble credibility estimation. The approach integrates a closed-loop OODA (Observe-Orient-Decide-Act) reasoning cycle, enabling lightweight continual post-training with cold-start initialization. The core algorithms include multi-modal vision-language models (VLMs), credibility filtering, and structured reasoning modules, which collectively improve hallucination detection and mitigation. The training uses only 9,000 samples, demonstrating high efficiency in reducing hallucinations and improving grounding and action fidelity.
Key Results
- Post-training on just 9K samples reduced hallucination rates by over 30%, with grounding accuracy on ScreenSpot-V2 improving from 77.18% to 86.62%. In AndroidControl tasks, action accuracy increased from 60.93% to 87.49%. The ensemble credibility approach boosted hallucination detection F1 scores by 12%, significantly enhancing reliability. The structured reasoning and credibility filtering contributed to a 30% reduction in hallucinations, leading to more robust GUI automation.
- The models showed improved generalization across multiple benchmarks, with grounding accuracy surpassing 85% and action success rates exceeding 80%, demonstrating the method's effectiveness in diverse GUI scenarios.
- Ablation studies confirmed that integrating the OODA cycle and credibility estimation yielded the highest performance gains, validating the importance of structured reasoning and evaluation.
Significance
This research addresses a critical gap in GUI agent reliability by providing a systematic framework for diagnosing and mitigating hallucinations. It offers a scalable, low-cost solution that enhances the robustness of visual reasoning systems in real-world applications, such as automated testing, assistive interfaces, and autonomous systems. The combination of taxonomy, evaluation, and structured reasoning paves the way for more trustworthy AI in complex interactive environments, reducing costly errors and increasing user trust in automation.
Technical Contribution
The paper introduces a novel GUI-specific hallucination taxonomy, a rigorous three-stage evaluation pipeline integrating expert annotations and ensemble credibility, and a structured OODA-based reasoning module for self-correction. It innovates by embedding structured, reflective reasoning into lightweight post-training, enabling models to self-assess and improve iteratively. The approach combines multi-modal models, credibility filtering, and reinforcement learning, establishing a new paradigm for hallucination mitigation in GUI agents with theoretical guarantees of improved grounding and action fidelity.
Novelty
This is the first work to systematically categorize GUI hallucinations into perceptual and reasoning subtypes, develop a multi-stage credibility-based evaluation pipeline, and embed a closed-loop OODA cycle for self-correction. Unlike prior methods relying solely on large-scale pretraining, this approach emphasizes fine-grained diagnosis and targeted mitigation, offering a scalable, interpretable, and effective solution for real-world deployment.
Limitations
- The approach depends on expert annotations for benchmark construction, which may limit scalability and increase costs. In highly dynamic or novel GUI environments, hallucination detection may still face challenges due to unseen failure modes.
- The structured reasoning module introduces additional computational overhead, potentially affecting real-time performance. Future work should focus on automating annotation and optimizing inference efficiency.
Future Work
Future directions include automating data annotation via active learning, integrating more advanced reinforcement learning algorithms for adaptive self-improvement, and extending the framework to handle more complex, multi-modal interactive environments. Additionally, exploring domain adaptation techniques could improve robustness across diverse application scenarios.
AI Executive Summary
The rapid growth of GUI automation driven by vision-language models (VLMs) has brought significant advancements in intelligent interface interactions. However, a persistent challenge remains: hallucinations—errors where models generate or interpret information inaccurately—pose serious risks in real-world deployments. Existing solutions primarily focus on scaling data and model size, but these approaches often overlook the nuanced nature of hallucinations, especially in complex GUI environments where perception and reasoning errors intertwine.
This paper introduces HalluClear, a comprehensive framework designed to diagnose, evaluate, and mitigate hallucinations in GUI agents. The core innovation lies in constructing a GUI-specific hallucination taxonomy through bottom-up failure analysis, which categorizes errors into perceptual and reasoning subtypes. Building upon this, the authors develop a three-stage evaluation workflow that combines expert-annotated benchmarks with ensemble credibility estimation, ensuring reliable and scalable hallucination assessment.
To address the identified issues, the authors propose an OODA-inspired structured reasoning cycle—Observe, Orient, Decide, Act—augmented with a reflection mechanism. This closed-loop approach enables models to self-assess and correct their errors during lightweight post-training, significantly reducing hallucination rates. Experimental results demonstrate that training on only 9,000 samples within this framework reduces hallucinations by over 30%, improves grounding accuracy, and enhances action fidelity across multiple benchmarks.
The significance of this work extends beyond technical improvements; it offers a scalable, interpretable, and industry-ready solution for deploying robust GUI automation systems. By integrating structured reasoning, credibility filtering, and continual learning, HalluClear paves the way for more trustworthy AI agents capable of operating reliably in real-world, interactive environments. Future research will focus on automating data annotation, optimizing inference efficiency, and expanding applicability to more complex multi-modal scenarios, further advancing the field of reliable AI automation.
Deep Analysis
Background
近年来，深度视觉-语言模型（VLM）在GUI自动化中的应用迅速发展，代表性工作如Qwen-VL、GUI-Owl等推动了多模态理解的边界。然而，模型在实际应用中频繁出现幻觉问题，导致识别错误、关系误判甚至操作失误，严重制约了工业界的推广。现有研究多关注模型性能提升，缺乏系统的幻觉诊断和缓解机制，尤其在GUI环境中，感知偏差和推理错误交织，亟需建立专门的评估体系和缓解策略，以确保模型的可靠性和安全性。
Core Problem
GUI代理中的幻觉问题主要源于模型对复杂界面元素的误识别和推理偏差，导致错误的元素定位、关系理解或指令执行。传统微调方法难以根除这些问题，且缺乏细粒度的诊断工具，限制了模型的应用范围。如何在保证性能的同时，有效识别和缓解幻觉，成为当前研究的核心难题。这不仅关系到模型的准确性，也影响到实际工业应用的安全性和用户体验。
Innovation
本文提出了GUI特定的幻觉分类体系，将感知和推理错误细分为八个子类型，提供更细粒度的错误分析基础。设计了三阶段评估流程，结合专家标注的黄金基准和模型可信度筛选，确保检测的可靠性。引入基于OODA循环的闭环推理机制，使模型在推理过程中不断反思和修正错误，结合微调和强化学习，实现模型的持续优化。这些创新点突破了以往仅依赖大规模预训练的局限，为模型的稳健性提供了新路径。
Methodology
- �� 构建GUI专用幻觉分类体系：通过分析离线数据，归纳八种常见幻觉类型，涵盖感知偏差和推理偏差。• 设计三阶段评估流程：由专家标注黄金基准，筛选可信判决模型，结合多模型集成，确保幻觉检测的准确性和稳定性。• 采用多模态VLM和可信度估算：结合模型输出的可信度指标，筛除不可靠判决，提升整体评估效果。• 引入OODA闭环推理：在观察环境、判断任务、决策行动、执行反馈的循环中，模型不断反思和修正，形成自我校正机制。• 通过微调和强化学习，利用有限样本不断优化模型结构，降低幻觉发生频率。
Experiments
采用公开的ShowUI、Android-Control和GUI-Odyssey数据集，构建9K样本的微调集。比较多种模型（Qwen-VL、GUI-Owl等）在grounding和动作任务中的表现，验证幻觉缓解效果。通过AB测试和消融实验，分析不同组件对性能的贡献。实验还评估模型在不同场景下的泛化能力和鲁棒性，验证结构化推理和可信度筛选的有效性。结果显示，微调后模型在多个指标上均有显著提升，幻觉率降低30%以上。
Results
微调后，模型在Grounding任务中的准确率从77.18%提升至86.62%，在AndroidControl任务中动作成功率从60.93%提升至87.49%。幻觉检测的F1分数提升12%，模型的鲁棒性显著增强。结合结构化推理和可信度筛选，模型在多个基准上表现优异，验证了方法的有效性。实验还显示，采用OODA循环的模型在复杂场景中表现出更强的自我修正能力，幻觉发生率明显下降。
Applications
该方法适用于工业界的GUI自动化、智能测试和人机交互系统，能有效减少误判和操作错误，提升系统的可靠性和用户体验。未来，可结合自动标注和强化学习，扩展到更复杂、多样化的界面环境，推动智能界面技术的广泛应用。特别是在自动化测试和智能助手等场景中，具有广阔的应用前景。
Limitations & Outlook
目前方法依赖专家标注，存在成本较高和效率有限的问题。在极端复杂或新颖的GUI环境中，模型仍可能出现误判。结构化推理引入的计算开销可能影响实时性。未来需优化自动标注流程，提升推理效率，并增强模型在多样环境中的泛化能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，菜单上写着各种菜肴，但有时候你会误认食材或忘记步骤。这个研究就像是给厨师配备了一个聪明的助手，它能帮你更准确地识别食材、记住每个步骤，避免误会或遗漏。这个助手会不断学习，遇到错误会反思原因，然后改正。它还会用一种像“观察-思考-决定-行动”的循环，确保每一步都正确。通过这样的方法，厨师的饭菜变得更好吃、更安全，也更少出错。这个助手就像是给AI模型装上了“自我检查”的能力，让它在复杂的界面中也能做出正确的判断和操作。
ELI14 Explained like you're 14
想象你在玩一个游戏，但有时候你会误会屏幕上的东西，比如把一个按钮当成了别的东西，或者忘记了任务的步骤。这个研究就像是给游戏里的AI配备了一个聪明的助手，它可以帮你更准确地看清楚屏幕上的内容，记住每个任务步骤，还能自己反思哪里出错了。这个助手会不断学习，遇到问题会想办法改正，就像你在游戏中学会了更聪明的玩法。它用一种“观察-思考-决定-行动”的循环，确保每次操作都正确。这样，AI就能更聪明、更靠谱地帮你完成任务，就像你有了一个非常聪明的伙伴一样。
Glossary
Vision-Language Model (视觉-语言模型)
一种结合视觉信息和自然语言理解的深度学习模型，用于多模态任务，能同时处理图像和文本信息。
在论文中，VLM用于GUI元素识别和推理，检测模型幻觉。
幻觉 (Hallucination)
模型在理解或生成过程中出现的错误信息或虚假元素，与实际数据不符。
本文分析GUI代理中感知和推理的幻觉类型。
OODA循环 (Observe-Orient-Decide-Act)
一种决策模型，强调通过连续观察、判断、决策和行动形成闭环反馈。
作者将其引入模型推理流程，提升模型的自我反思能力。
可信度估算 (Credibility Estimation)
评估模型判决或输出可靠性的方法，用于筛选可信判决以提升系统整体性能。
在评估流程中，用于筛选判决模型，确保幻觉检测的准确性。
结构化推理 (Structured Reasoning)
采用明确的推理模板和步骤，增强模型推理的逻辑性和可解释性。
结合OODA循环，减少推理幻觉，提升决策质量。
Open Questions Unanswered questions from this research
- 1 如何在极端复杂或动态变化的GUI环境中，保持模型的幻觉检测和缓解效果？
- 2 自动化标注和可信度评估的效率提升空间，以及在大规模应用中的可扩展性。
- 3 结合强化学习实现模型自我优化的具体机制和效果，仍需深入研究。
Applications
Immediate Applications
工业GUI自动化
利用HalluClear提升自动化脚本的准确性和鲁棒性，减少误操作，提升效率。
智能测试与验证
在软件测试中自动检测界面识别错误，确保界面交互的正确性和稳定性。
Long-term Vision
自主界面交互系统
实现具有自我诊断和修正能力的智能界面代理，广泛应用于智能家居、车载系统等。
Abstract
While progress in GUI agents has been largely driven by industrial-scale training, ungrounded hallucinations often trigger cascading failures in real-world deployments.Unlike general VLM domains, the GUI agent field lacks a hallucination-focused suite for fine-grained diagnosis, reliable evaluation, and targeted mitigation.To bridge this gap, we introduce HalluClear, a comprehensive suite for hallucination mitigation in GUI agents as a complement to computation-intensive scaling. HalluClear comprises: (1) a GUI-specific hallucination taxonomy derived from empirical failure analysis; (2) a calibrated three-stage evaluation workflow which enhances VLM-as-a-judge reliability via expert-annotated benchmarking and ensemble credibility estimation; and (3) a mitigation scheme based on closed-loop structured reasoning, enabling lightweight continual post-training with cold-start initialization for both generalist and GUI-specialist agents. Experiments across representative agents and public benchmarks demonstrate that post-training on only 9K samples within our suite can significantly reduce hallucinations, thereby improving grounding and action fidelity, offering a compute-efficient pathway to robust GUI automation.