What Happens Before Decoding? Prefill Determines GUI Grounding in VLMs
Re-Prefill introduces attention-guided second prefill to improve GUI element localization, boosting accuracy up to 4.3%.
Key Findings
Methodology
This paper analyzes the inference process of VLMs in GUI grounding, revealing that the prefill stage determines candidate selection, which is difficult to correct later. It proposes an attention-guided second prefill strategy, leveraging cross-layer high-attention visual tokens to re-evaluate candidate regions. This approach combines the instruction embedding with selected visual tokens, enabling the model to reconsider its target before coordinate decoding. Extensive experiments across four models and five benchmarks demonstrate consistent improvements, with gains up to 4.3%. The core innovation lies in exploiting layer-wise attention consistency to identify key visual tokens, guiding a second prefill that refines target selection without additional training.
Key Results
- Across four VLMs, Re-Prefill improves performance on five GUI benchmarks, with the highest gain of 4.3% on ScreenSpot-Pro, increasing from 65.8% to 70.1%. On OSWorld-G, it boosts accuracy by 2.7%, from 63.0% to 70.1%. The method outperforms existing training-free approaches like ZoomClick and UI-Zoomer, validating its effectiveness and generalization.
- In complex interaction scenarios, Re-Prefill remains effective across platforms and instruction types, reducing target localization errors. The attention-based candidate filtering significantly narrows the candidate space, leading to more accurate final coordinates.
- By identifying visually salient tokens via cross-layer attention consistency, and applying a layer-wise prefix injection, the method enhances the model’s focus on correct regions. This approach demonstrates robustness across different model sizes and datasets, indicating broad applicability.
Significance
This work uncovers the pivotal role of the prefill stage in GUI grounding, addressing the longstanding challenge of error correction during autoregressive decoding. The attention-guided second prefill offers a simple yet powerful inference-time enhancement, improving accuracy without retraining. Its broad applicability across models and benchmarks signifies a step forward in zero-shot GUI understanding, facilitating more reliable autonomous agents and intelligent interfaces. The approach bridges the gap between model internal attention dynamics and practical localization tasks, opening new avenues for research in visual-language reasoning.
Technical Contribution
The paper introduces a novel, training-free second prefill mechanism guided by cross-layer attention consistency, enabling dynamic re-evaluation of candidate regions. This mechanism transitions from full-context to focused visual conditioning via layer-wise prefix injection, preserving semantic alignment while refining target localization. The method leverages internal attention signals, avoiding additional training or fine-tuning, and enhances autoregressive coordinate decoding robustness. Extensive experiments validate its effectiveness across diverse models and datasets, establishing a new paradigm for inference-time optimization in VLMs.
Novelty
This is the first work to explicitly analyze and leverage the internal attention dynamics during prefill in GUI grounding, proposing a layer-wise attention consistency filter for candidate region re-evaluation. Unlike prior multi-pass or post-processing methods, Re-Prefill dynamically refines target selection within the inference process itself, offering a fundamentally different approach to error correction. Its innovative use of internal attention signals for self-guided re-evaluation marks a significant advance in zero-shot visual-language reasoning.
Limitations
- The method relies on stable and interpretable attention signals; in scenarios with highly ambiguous or occluded UI elements, the attention-based filtering may fail, reducing effectiveness.
- Additional computational overhead from the second prefill step may impact real-time applications, especially on large-scale models or resource-constrained devices.
- Currently, the approach does not incorporate multi-modal cues beyond visual tokens and instructions; integrating other modalities could further improve robustness.
Future Work
Future directions include integrating multi-modal cues such as user interaction logs or contextual cues to further refine candidate selection. Developing adaptive thresholding mechanisms for attention filtering could improve robustness across diverse scenarios. Extending the approach to multi-turn dialogues and dynamic interfaces will be explored, aiming for more interactive and context-aware GUI understanding. Additionally, efforts to reduce computational costs and improve explainability of attention signals will be prioritized, facilitating deployment in real-world systems.
AI Executive Summary
This study investigates the inference process of Vision-Language Models (VLMs) in GUI element localization, revealing that the prefill stage plays a decisive role in candidate element selection. Traditional approaches often rely on multiple inference passes, such as iterative cropping or candidate aggregation, which are computationally intensive and do not allow interaction between visual tokens across passes. The authors analyze the internal attention dynamics, discovering that attention during prefill is dispersed over multiple regions, establishing a coarse candidate scope, and then sharply concentrates during decoding to generate precise coordinates. However, errors made during prefill tend to persist, as they are rarely corrected during autoregressive decoding. To address this, the paper proposes Re-Prefill, a training-free method that leverages cross-layer attention consistency to identify key visual tokens representing candidate regions. These tokens, along with instruction embeddings, are used in a second prefill step, enabling the model to re-evaluate and refine its target selection before coordinate generation. Extensive experiments across four models and five benchmarks, including high-resolution interfaces and complex interactions, demonstrate that Re-Prefill consistently improves accuracy, with gains up to 4.3%. The approach outperforms existing inference-time methods, offering a simple yet effective solution that enhances GUI grounding without additional training. This work shifts the focus to the importance of the prefill stage, providing new insights into model internal mechanisms and opening pathways for future research in zero-shot visual-language reasoning and autonomous interface understanding. Overall, Re-Prefill exemplifies how internal attention signals can be exploited to optimize inference, paving the way for more reliable and efficient AI systems in real-world applications.
Deep Dive
Abstract
Existing training-free approaches for GUI grounding often rely on multiple inference runs, such as iterative cropping or candidate aggregation, to identify target elements. Despite this additional computation, each forward pass still independently interprets the instruction and parses the visual layout, without enabling progressive interaction among visual tokens. In this paper, we study what happens during GUI grounding in Vision-Language Models (VLMs) and identify a previously overlooked bottleneck. We show that grounding follows a two-stage paradigm: the prefill stage determines candidate UI elements, while the decoding stage subsequently refines the final coordinates. This asymmetry establishes prefill as the critical step, as errors in candidate selection cannot be effectively corrected during decoding. Based on this observation, we propose Re-Prefill, a training-free method that revisits inference by introducing an attention-guided second prefill stage to refine target selection. Specifically, visual tokens that consistently receive high attention from the query position, i.e., the final token, across layers are extracted as a preliminary target hypothesis and appended to the input, together with the instruction hidden states, enabling the model to deeply re-think its decision before coordinate generation. Experiments across four VLMs and five benchmarks, including ScreenSpot-Pro, ScreenSpot-V2, OSWorld-G, UI-Vision, and MMBench-GUI, demonstrate consistent improvements without additional training, with gains of up to 4.3% on ScreenSpot-Pro. Code will be available at https://github.com/linjiaping1/Re-Prefill.