DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoning

Recent advances in multimodal large language models (MLLMs) like BLIP and CLIP have propelled GUI understanding toward more intelligent automation. Early methods relied on structured data such as DOM trees or OCR detection, but these approaches faced limitations in handling complex, high-resolution interfaces with cluttered layouts. Recent models have improved element localization but still struggle with visual ambiguity, modality imbalance, and computational efficiency. Despite progress, achieving robust, real-time GUI comprehension in diverse environments remains an open challenge, especially without extensive training data or annotations.

The core challenge lies in accurately localizing small, densely packed GUI elements in high-resolution screens. Traditional methods are hampered by visual redundancy, clutter, and the imbalance between textual and icon modalities. Existing models tend to over-rely on text, neglecting icon cues, which leads to errors in ambiguous or cluttered scenes. Moreover, high-resolution images cause computational bottlenecks, and single-pass inference often fails to resolve fine-grained details, resulting in low accuracy and robustness. Addressing these issues requires a scalable, training-free approach capable of adaptive, multi-modal reasoning.

The paper introduces three key innovations: 1) Modality decoupling, which separates text and icon processing to reduce cross-modal interference; 2) Dynamic zooming, which iteratively crops and refines target regions based on initial predictions, improving localization in high-resolution images; 3) Dynamic halting, which assesses the stability of predictions to stop zooming early, balancing accuracy and efficiency. This combination enables high-precision, robust GUI grounding without additional training, making it adaptable and easy to deploy across diverse models and scenarios.

• �� Divide GUI elements into text and icon modalities, process each with pretrained vision-language models.
• �� Use initial coarse predictions to crop the image around the target, then iteratively zoom in, refining the bounding box.
• �� Implement a dynamic stop condition based on the spatial change of predicted coordinates, halting zooming when the target is sufficiently localized.
• �� At each iteration, crop the image to focus on the predicted region, reducing visual noise.
• �� Combine modality-specific predictions to select the final target, improving robustness.
• �� The entire process is modular, leveraging existing models, and requires no additional training, making it highly scalable.

Experiments were conducted on ScreenSpot and ScreenSpot-Pro datasets, evaluating IoU and localization accuracy. Baseline models included OS-Atlas-7B and UGround-V1-7B. The framework was integrated without retraining, and ablation studies tested the effects of each component. Results showed performance gains of over 2x, with IoU scores exceeding 65% in complex scenes. The experiments validated the effectiveness of modality decoupling and dynamic zooming across different interface types and resolutions.

The integration of DiMo-GUI improved IoU from ~30% to over 65% in high-resolution scenarios, with performance gains consistent across multiple models. Modal decoupling increased icon recognition accuracy by 12%, while dynamic zooming reduced false positives and improved localization precision. The adaptive iteration mechanism prevented overfitting and unnecessary computation, achieving a balance between speed and accuracy. Overall, the framework demonstrated strong generalizability and robustness, significantly outperforming existing single-pass methods.

This approach is suitable for automating GUI testing, accessibility tools, and intelligent interface assistants. Its plug-and-play nature allows deployment in various platforms—mobile, web, desktop—without retraining. The method enhances real-time interaction, enabling more reliable and precise element recognition, crucial for industry applications like automated QA, UI design, and assistive technologies.

Despite its strengths, the framework's initial coarse predictions can be inaccurate in extremely cluttered or occluded environments, limiting subsequent refinement. Its reliance on pretrained models constrains performance in domain-specific interfaces without further adaptation. Additionally, in multi-object scenarios, the current approach may need extension to handle multiple targets simultaneously, and real-time deployment requires further optimization.