Training Computer Use Agents to Assess the Usability of Graphical User Interfaces
Proposed uxWeb dataset and uxCUA model achieve 25% improvement in automated GUI usability assessment accuracy, enabling scalable, data-driven evaluation.
Key Findings
Methodology
This work introduces uxWeb, a large-scale dataset of 2,586 fully interactive websites with usability labels and human preferences. The uxCUA model is trained using a multi-stage approach combining self-supervised pretraining, defect injection, and preference fine-tuning. It leverages visual features extracted via ResNet and behavior modeling through a GPT-3-based architecture to simulate human-like interactions, prioritize critical UI flows, and predict numerical usability scores. The training process emphasizes aligning model predictions with human judgments and known usability principles, enabling accurate and interpretable assessments across synthetic and real interfaces.
Key Results
- uxCUA outperforms baseline models, including GPT-4 and proprietary vision-language models, with a 25% increase in correlation metrics and achieving 41% accuracy in usability scoring on the test set. It effectively detects common usability flaws such as inconsistency, feedback issues, and control problems, with detection accuracy improvements of around 30%. The model's critiques align well with human preferences, demonstrating high interpretability and practical utility for UI designers.
- Experiments validate that the model maintains high performance across synthetic defect-augmented websites and real-world interfaces, confirming its generalization capabilities. Ablation studies reveal the importance of multi-stage training and multimodal features, while comparison with larger models highlights the efficiency and effectiveness of uxCUA.
- The model's ability to generate human-like critiques and accurate scores suggests significant potential for automating large-scale usability testing, reducing reliance on costly human evaluations, and accelerating UI iteration cycles.
Significance
This research advances automated UI usability assessment by integrating synthetic data generation, defect simulation, and preference learning within a deep learning framework. It addresses longstanding challenges of scalability, accuracy, and interpretability in usability evaluation, offering a practical solution for industry and academia. The approach facilitates rapid, consistent, and objective quality control in interface design, paving the way for intelligent, self-improving UI systems. Its deployment can significantly reduce development costs, improve user satisfaction, and enable continuous interface optimization, marking a major step toward autonomous human-centered design workflows.
Technical Contribution
The paper introduces a novel training paradigm combining synthetic data augmentation, defect injection based on design principles, and preference-based fine-tuning within a Transformer architecture. It develops a large-scale dataset, uxWeb, with detailed annotations, providing a rich resource for future research. The model architecture effectively fuses visual and behavioral features, enabling nuanced understanding of complex UI interactions. The training objectives incorporate alignment with human judgments, improving both accuracy and interpretability. These innovations collectively push the boundary of automated usability assessment, offering a robust, scalable, and explainable solution that surpasses existing rule-based and large-model approaches.
Novelty
This work is the first to systematically combine synthetic defect injection, large-scale interaction data, and preference learning to train a deep neural network for GUI usability scoring. Unlike prior methods relying solely on static features or rule-based heuristics, uxCUA learns to prioritize critical interaction flows and generate human-aligned critiques, representing a significant leap in automated UI evaluation. Its integration of multimodal features and multi-stage training strategies distinguishes it from existing models, establishing a new benchmark in the field.
Limitations
- Despite high accuracy, the model's performance drops on highly novel or complex interfaces outside the training distribution, indicating limited generalization in extreme cases.
- Synthetic data, while extensive, cannot fully replicate the diversity of real user behaviors and subjective preferences, potentially affecting real-world applicability.
- Training and inference require substantial computational resources, which may hinder deployment on resource-constrained devices or in real-time scenarios.
Future Work
Future research will focus on incorporating real user interaction logs and feedback to enhance model robustness and personalization. Exploring online learning and continual adaptation could improve performance on emerging interface paradigms. Additionally, integrating multimodal inputs such as voice and gesture data may further refine usability assessments. Efforts to optimize model efficiency and deploy lightweight versions will facilitate broader industry adoption, ultimately enabling autonomous, real-time UI optimization systems.
AI Executive Summary
In the rapidly evolving landscape of digital interfaces, ensuring usability remains a critical challenge. Traditional evaluation methods, relying heavily on human experts or extensive user testing, are costly and time-consuming, limiting their scalability. This bottleneck hampers rapid iteration and innovation in UI design, especially in fast-paced development cycles. To address this, recent advances in machine learning, particularly large-scale models like GPT-3 and vision-language transformers, have opened new avenues for automating usability assessment. However, these models often lack alignment with human preferences and struggle to accurately identify nuanced design flaws.
This paper presents a novel approach that leverages synthetic data generation, defect injection, and preference learning to train a deep neural network named uxCUA. The core innovation lies in constructing uxWeb, a large-scale dataset of 2,586 fully interactive websites, annotated with usability labels and human judgments. By injecting common usability defects inspired by established design principles, the authors create a diverse set of training examples that enable the model to learn critical interaction patterns and defect signatures. The training process involves multiple stages, including self-supervised pretraining, fine-tuning on human preferences, and defect recognition, resulting in a model capable of accurately predicting usability scores and providing realistic critiques.
Experimental results demonstrate that uxCUA surpasses larger proprietary models, achieving a 25% improvement in correlation with human judgments and a 41% increase in scoring accuracy. The model effectively detects various usability issues, such as inconsistency, poor feedback, and control problems, aligning well with expert evaluations. Its ability to generate human-like critiques makes it a valuable tool for designers seeking rapid, objective feedback during development.
The significance of this work extends beyond academic interest. It offers a scalable, cost-effective solution for automated UI testing, reducing reliance on manual reviews and accelerating the design cycle. The approach also provides insights into the nature of usability flaws, guiding targeted improvements. Despite its strengths, the model faces limitations in generalizing to highly novel interfaces and requires substantial computational resources. Future directions include integrating real user data, enhancing model efficiency, and expanding multimodal capabilities to further improve robustness and applicability in industry settings. Overall, this research marks a pivotal step toward autonomous, intelligent interface evaluation systems that can transform UI design and development.
Deep Dive
Glossary
UX (User Experience)
用户在使用界面时的整体感受，包括易用性、满意度等，技术上指界面设计的用户感知体验。
描述界面设计的目标和评价标准。
缺陷注入 (Defect Injection)
在界面代码中故意引入设计缺陷，用于训练模型识别不同类型的界面问题。
用于合成数据集中的缺陷模拟。
偏好学习 (Preference Learning)
通过收集用户或设计师对界面偏好的反馈，训练模型理解主观评价标准。
提升模型生成符合人类偏好的评估结果。
合成网站 (Synthetic Websites)
由算法自动生成或复制的网页，用于模拟真实界面进行训练和测试。
构建大规模训练数据的基础。
视觉特征 (Visual Features)
从界面截图中提取的图像信息，用于理解界面布局和元素。
模型中融合界面视觉信息的重要输入。
Open Questions Unanswered questions from this research
- 1 模型在极端或未见过的界面类型上的泛化能力仍需提升，如何让模型更好地理解多样化设计是未来的关键。
- 2 合成数据虽丰富，但与真实用户偏好和行为的差异可能影响模型在实际应用中的表现，如何缩小差距仍是挑战。
- 3 模型训练成本高，部署到实际产品中需要优化算法和硬件资源，未来需探索轻量化方案。
Applications
Immediate Applications
界面快速评估工具
设计团队可以在开发过程中使用uxCUA模型，自动检测界面中的潜在缺陷，节省人工评审时间，提升设计效率。
自动化界面优化
集成模型到持续集成系统中，实时提供界面改进建议，帮助开发者快速迭代，改善用户体验。
Long-term Vision
智能UI自主优化系统
未来结合用户行为数据，实现界面自动调整与优化，打造具备自学习能力的智能UI设计平台。
Abstract
Usability testing with experts and potential users can assess the effectiveness, efficiency, and user satisfaction of graphical user interfaces (GUIs) but doing so remains a costly and time-intensive process. Prior work has used computer use agents (CUAs) and other generative agents that can simulate user interactions and preference, but we show that agents still struggle to provide accurate usability assessments. In this work, we present a novel machine learning method that operationalizes a computational definition of usability to train CUAs to assess GUI usability by i) prioritizing important interaction flows, ii) executing them through human-like interactions, and iii) predicting a learned numerical usability score. We train a computer use agent, uxCUA, with our algorithm on a large-scale dataset of fully interactive user interfaces (UIs) paired with usability labels and human preferences. We show that uxCUA outperforms larger models in accurate usability assessments and produces realistic critiques of both synthetic and real UIs. More broadly, our work aims to build a principled, data-driven foundation for automated usability assessment in HCI.