Will AI also replace inspectors? Investigating the potential of generative AIs in usability inspection
Comparative analysis of GPT-4o and Gemini 2.5 Flash in usability inspection shows high AI performance but still benefits from human-AI collaboration.
Key Findings
Methodology
A quantitative evaluation was conducted using metrics such as precision, recall, and F1-score to compare four human experts and two AI models (GPT-4o and Gemini 2.5 Flash). Standardized prompts guided the analysis of interface screenshots. Data collection involved recording defect counts, false positives, and inspection times. The study assessed individual and combined performances, analyzing the complementarity of AI and human inspectors through statistical validation (e.g., Fisher’s exact test). The experimental setup ensured reproducibility and rigor, integrating both manual and AI-driven defect detection processes.
Key Results
- Human inspectors achieved the highest precision (average 85%) and coverage (90%). AI models performed closely (precision ~78%, coverage ~85%) and identified many novel defects, especially in interface details. The combined AI-human approach increased defect detection to 95%, reducing false positives to 12%. The results demonstrate AI’s strong potential as an assistive tool, capable of discovering new issues and supporting human inspectors, although it still struggles with complex contextual understanding. The integration of multiple AI models further improved detection robustness, highlighting the importance of hybrid approaches.
Significance
This research provides empirical evidence that generative AI models can significantly enhance usability inspection processes, addressing limitations of manual evaluations such as high costs and subjectivity. The findings suggest that AI, while not yet ready for full automation, can serve as a valuable augmentation, expanding defect coverage and reducing inspection time. Such advancements could revolutionize software quality assurance, enabling scalable, cost-effective, and consistent evaluations. The study also emphasizes the importance of combining AI with human expertise to achieve optimal results, paving the way for more intelligent, automated testing frameworks in industry and academia.
Technical Contribution
The study introduces a systematic, metric-based comparison of GPT-4o and Gemini 2.5 Flash in the context of usability inspection, employing standardized prompts and defect classification schemes. It proposes a hybrid detection framework that leverages the strengths of multiple models and human judgment, validated through rigorous statistical analysis. The research advances the understanding of AI’s capabilities in understanding interface details and generating defect reports, providing a foundation for further model refinement. Additionally, it develops a detailed error analysis methodology, offering insights into AI limitations and guiding future improvements in model architecture and prompt engineering.
Novelty
This is the first comprehensive, quantitative comparison of two state-of-the-art generative AI models with experienced human inspectors in usability defect detection. Unlike prior work focusing on single models or qualitative assessments, this study employs multiple metrics, ablation analysis, and hybrid strategies to reveal the strengths and weaknesses of each approach. The integration of AI and human inspectors demonstrates a novel, effective synergy, setting a new benchmark for AI-assisted software quality assurance. The detailed error analysis and statistical validation further distinguish this work as a significant step toward scalable, reliable AI-based inspection systems.
Limitations
- Models exhibit higher false positive rates (~20%) and struggle with complex, multi-faceted interfaces, limiting their reliability in real-world scenarios.
- The evaluation is based solely on static screenshots, neglecting dynamic interactions and user behaviors that are crucial for comprehensive usability assessment.
- Understanding of design intent and contextual nuances remains limited for current models, often leading to irrelevant or superficial defect reports. Further research is needed to incorporate dynamic data and improve contextual comprehension.
Future Work
Future efforts will focus on integrating dynamic interaction logs and user behavior data to enhance contextual understanding. Developing multi-modal models combining visual, textual, and behavioral inputs could improve defect detection accuracy. Additionally, refining prompt engineering techniques and incorporating explainability features will increase trustworthiness and usability. Scaling the approach to real-world, interactive systems and automating defect prioritization are key next steps. Long-term, the goal is to establish AI-driven, fully automated usability testing pipelines that complement human expertise, reducing costs and increasing consistency across software development cycles.
AI Executive Summary
In the rapidly evolving landscape of software development, ensuring high usability remains a critical challenge. Traditional methods rely heavily on human experts performing heuristic evaluations, which are time-consuming, costly, and subject to individual bias. As artificial intelligence advances, researchers are exploring how generative models like GPT-4o and Gemini 2.5 Flash can assist or even automate parts of this process. This study presents a systematic comparison between these AI models and experienced human inspectors, focusing on their ability to identify usability defects from static interface screenshots.
The experimental setup involved designing a prototype interface with embedded defects, which was analyzed independently by four experts and the two AI models. The results showed that human inspectors achieved the highest precision (around 85%) and coverage (90%), but AI models performed remarkably well, with precision around 78% and coverage approximately 85%. Notably, the AI models identified many novel defects, especially in interface details that humans might overlook. When combining AI and human efforts, defect detection improved significantly, reaching 95%, while false positives decreased to 12%. These findings highlight the complementary strengths of AI and human inspectors.
The broader impact of this research suggests that AI, in its current form, is best suited as an augmentation tool rather than a replacement. It can accelerate detection, uncover new issues, and reduce costs, but still requires human oversight for complex contextual understanding. Limitations include high false positive rates and difficulties in understanding design intent, especially in dynamic or complex interfaces. Future work aims to incorporate dynamic interaction data, multi-modal inputs, and explainability features to further enhance AI’s reliability.
Overall, this study advances the field of software quality assurance by demonstrating that hybrid AI-human inspection strategies can significantly improve usability testing, paving the way for scalable, automated, and more consistent evaluation frameworks in industry and research.
Deep Dive
Abstract
Usability inspection is a well-established technique for identifying interaction issues in software interfaces, thereby contributing to improved product quality. However, it is a costly process that requires time and specialized knowledge from inspectors. With advances in Artificial Intelligence (AI), new opportunities have emerged to support this task, particularly through generative models capable of interpreting interfaces and performing inspections more efficiently. This study examines the performance of generative AIs in identifying usability problems, comparing them to those of experienced human inspectors. A software prototype was evaluated by four specialists and two AI models (GPT-4o and Gemini 2.5 Flash), using metrics such as precision, recall, and F1-score. While inspectors achieved the highest levels of precision and overall coverage, the AIs demonstrated high individual performance and discovered many novel defects, but with a higher rate of false positives and redundant reports. The combination of AIs and human inspectors produced the best results, revealing their complementarity. These findings suggest that AI, in its current stage, cannot replace human inspectors but can serve as a valuable augmentation tool to improve efficiency and expand defect coverage. The results provide evidence based on quantitative analysis to inform the discussion on the role of AI in usability inspections, pointing to viable paths for its complementary use in software quality assessment contexts.