LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization

Key Findings

Methodology

LPO integrates information entropy to predict potential interaction zones, employing grid-based window segmentation. It introduces a dynamic reward based on Euclidean distance to emphasize spatial accuracy, and leverages GRPO for global preference exploration. The approach involves: • Dividing the interface into non-overlapping windows; • Computing entropy for each window to estimate information richness; • Using maximum entropy windows for position prediction; • Applying a distance-based reward to refine localization; • Combining these via GRPO to guide policy optimization. This synergy enhances interaction precision and exploration efficiency.

Key Results

On offline benchmarks like Mind2Web (Deng et al., 2023b) and VisualWebBench (Liu et al., 2024), LPO outperforms SOTA methods with 3-5% improvements in element accuracy, operation F1, and step success rate. Online, on WebVoyager (He et al., 2024), task success rate increased by an average of 9%. Ablation studies confirm that both entropy and distance rewards contribute significantly, with combined effects yielding the best performance. The model demonstrates robustness across diverse tasks and environments, especially in real-world scenarios.
Compared to static boundary or text-matching strategies, LPO achieves higher localization accuracy, especially in complex, dynamic GUIs. Its ability to generalize across multiple domains and scenarios underscores its practical value. The results validate the effectiveness of integrating information density and spatial distance in reward design, setting new benchmarks for GUI agent precision.
The experimental data indicate that LPO significantly reduces positional errors, improves interaction success, and enhances exploration in large GUI spaces. Its superior performance in both offline and online tests highlights its potential for real-world deployment, with promising scalability and adaptability.

Significance

This work addresses a fundamental bottleneck in autonomous GUI interaction—precise spatial localization. By combining information density with dynamic spatial rewards, LPO offers a robust solution that surpasses traditional static boundary or text-based methods. Its capacity to explore large, complex environments efficiently makes it highly relevant for automating web and app interactions, virtual assistants, and human-computer interfaces. The approach paves the way for more intelligent, adaptable, and precise autonomous agents, reducing manual effort and increasing operational efficiency. Its implications extend to industries like automation, accessibility, and user experience design, where high-precision interaction is crucial.

Technical Contribution

LPO introduces a novel reward mechanism that fuses information entropy-based region importance with a dynamic, distance-sensitive reward function. This design allows the model to focus on information-rich zones while maintaining spatial accuracy. The integration of GRPO enhances exploration by comparing relative preferences, enabling the agent to learn more effective policies in expansive GUI spaces. The framework is built upon end-to-end deep learning, leveraging visual and positional cues to improve localization and decision-making. These innovations collectively outperform existing static boundary and text-matching strategies, providing a new theoretical and practical foundation for high-precision GUI interaction.

Novelty

This research is the first to utilize information entropy for predicting interaction zones in GUI interfaces, combined with a dynamic, distance-based reward to refine spatial localization. Unlike prior static boundary or text-matching methods, LPO adaptively emphasizes information-rich regions and spatial accuracy. Its integration with GRPO for global preference exploration is a novel approach that significantly enhances exploration efficiency and interaction precision, setting a new paradigm in autonomous GUI agent design.

Limitations

The method relies on fixed grid segmentation parameters, which may not adapt well to highly dynamic or irregular interfaces, requiring further adaptive mechanisms.
Training and inference involve substantial computational costs, especially for large-scale environments, limiting real-time deployment.
The entropy and distance rewards can be sensitive to noise or ambiguous visual signals, potentially affecting localization robustness. Future work should focus on robustness and efficiency improvements.

Future Work

Future directions include integrating multi-modal cues such as textual context and user behavior data to further improve localization accuracy. Developing adaptive grid segmentation and lightweight models will enhance real-time performance. Expanding evaluations to more diverse, real-world scenarios will validate robustness and scalability. Additionally, exploring unsupervised or semi-supervised learning paradigms could reduce data dependency, accelerating deployment in practical applications.

AI Executive Summary

The rapid evolution of autonomous agents has transformed how humans interact with graphical user interfaces (GUIs). Traditional methods, primarily based on supervised fine-tuning, struggle with precise spatial localization due to limited perception of positional data. Reinforcement learning approaches have attempted to address this but often lack effective mechanisms to evaluate positional accuracy, resulting in suboptimal performance. This gap hampers the deployment of reliable, high-precision GUI agents essential for automation and intelligent assistance.

In response, this paper introduces Location Preference Optimization (LPO), a novel framework that leverages information entropy and dynamic spatial rewards to enhance interaction accuracy. The core idea involves dividing the interface into grids, calculating the information density of each region, and predicting interaction zones based on maximum entropy. Simultaneously, a dynamic reward based on Euclidean distance guides the agent to refine its spatial precision. The integration of Group Relative Preference Optimization (GRPO) further enables the agent to explore large GUI spaces effectively, optimizing preferences for accurate interactions.

Extensive experiments across offline benchmarks like Mind2Web and VisualWebBench, as well as online evaluations on WebVoyager, demonstrate LPO’s superior performance. The method achieves 3-5% improvements in element recognition and step success rates in offline tests, and a 9% increase in task success rate online. Ablation studies confirm the individual contributions of entropy and distance rewards, with their combination yielding the best results. These findings highlight the potential of LPO to set new standards in autonomous GUI interaction, addressing longstanding challenges in spatial localization.

The significance of this work lies in its innovative fusion of information theory and spatial rewards, providing a robust, scalable solution for high-precision automation. It opens pathways for deploying autonomous agents in complex, real-world environments, such as web automation, virtual assistants, and accessibility tools. Despite current limitations in computational cost and interface adaptability, future work aims to incorporate multi-modal signals, optimize efficiency, and extend real-world validation. Overall, LPO marks a pivotal step toward intelligent, precise, and scalable GUI agents, promising transformative impacts across industry and research domains.

Deep Dive

Plain Language Accessible to non-experts

想象你在一个超级复杂的厨房里做菜。每个区域都可能藏着你需要的食材，但有些地方信息丰富，比如冰箱旁边或调料架上，而空旷的桌子上几乎没有用的东西。你需要找到最重要的地方去拿东西，不能随便乱跑。这个研究就像教厨师用一种聪明的方法，先用“信息密度”判断哪里可能藏有食材，然后用“距离”判断哪个地方更容易到达。这样，厨师就能更快、更准地找到需要的东西，做出美味的菜。这个方法让自动机器人在界面上也能像厨师一样聪明，知道在哪个区域操作最有效，极大提高了效率和准确性。

ELI14 Explained like you're 14

想象你在玩一个超级复杂的游戏，你需要找到特定的宝藏。游戏里的地图上有很多地方，但宝藏通常藏在信息丰富的区域，比如宝箱旁边或任务提示附近。而空旷的地方几乎没有用。这个研究就像教你用一种聪明的方法，先找出地图上信息最多的区域，然后根据距离判断哪个区域更容易到达。这样，你就可以更快找到宝藏，不会迷路。这就像给机器人装上了“聪明的眼睛”和“导航系统”，让它在界面上也能像你一样聪明地找到目标，操作更准确、更快。

Glossary

Information Entropy (信息熵)

衡量界面区域信息丰富程度的指标，越高代表区域越复杂、内容越多。

用于预测潜在交互位置的区域重要性。

Dynamic Distance Reward (动态距离奖励)

根据操作点与目标点的空间距离动态调整奖励值，鼓励空间定位的精确性。

提升GUI交互中位置的空间精度。

GRPO (Group Relative Preference Optimization)

一种偏好优化框架，通过比较偏好优势值，指导策略探索。

实现偏好引导的全局空间探索。

GUI Grounding (GUI定位)

将界面元素与视觉信息对应的能力，确保操作的准确性。

评估模型在界面元素上的空间定位效果。

Preference Optimization (偏好优化)

通过偏好信号引导模型学习更优的交互策略。

提升GUI代理的交互精度与效率。

Open Questions Unanswered questions from this research

1 如何在极端复杂或动态变化的界面中保持高精度定位仍是未解难题，现有模型在快速变化环境中的鲁棒性不足，未来需要结合多模态信息和自适应机制提升性能。

Applications

Immediate Applications

Web Automation

利用LPO实现网页元素的高精度识别与自动操作，适用于自动填写、数据采集等场景。

Intelligent Interface Assistants

为虚拟助手提供更准确的界面交互能力，提升用户体验和操作效率。

Long-term Vision

Autonomous Interface Systems

结合多模态信息，开发全自动、适应多场景的智能交互系统，降低人工成本。

Abstract

The advent of autonomous agents is transforming interactions with Graphical User Interfaces (GUIs) by employing natural language as a powerful intermediary. Despite the predominance of Supervised Fine-Tuning (SFT) methods in current GUI agents for achieving spatial localization, these methods face substantial challenges due to their limited capacity to accurately perceive positional data. Existing strategies, such as reinforcement learning, often fail to assess positional accuracy effectively, thereby restricting their utility. In response, we introduce Location Preference Optimization (LPO), a novel approach that leverages locational data to optimize interaction preferences. LPO uses information entropy to predict interaction positions by focusing on zones rich in information. Besides, it further introduces a dynamic location reward function based on physical distance, reflecting the varying importance of interaction positions. Supported by Group Relative Preference Optimization (GRPO), LPO facilitates an extensive exploration of GUI environments and significantly enhances interaction precision. Comprehensive experiments demonstrate LPO's superior performance, achieving SOTA results across both offline benchmarks and real-world online evaluations. Our code will be made publicly available soon, at https://github.com/jqtangust/LPO.

cs.LG cs.AI cs.CV