Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Ferret-UI enhances mobile UI understanding using multimodal LLMs, surpassing GPT-4V.
Key Findings
Methodology
Ferret-UI employs multimodal LLMs with 'any resolution' technology, dividing screens into sub-images to enhance visual features. It uses training samples for instruction following and region annotation to improve UI screen comprehension.
Key Results
- Ferret-UI surpasses GPT-4V in elementary UI tasks, particularly in icon recognition and text finding tasks, with a 15% accuracy improvement.
- In advanced tasks, Ferret-UI shows excellent reasoning ability, especially in function inference tasks, outperforming Fuyu and CogAgent by 20%.
- In the Spotlight benchmark, Ferret-UI excels in screen summarization and widget description tasks, with CIDEr scores of 115.6 and 140.3, respectively.
Significance
This research advances the application of multimodal LLMs in mobile apps by enhancing UI screen understanding, addressing existing models' limitations in UI interaction, and opening new possibilities for automation and accessibility.
Technical Contribution
Ferret-UI integrates 'any resolution' technology with multimodal LLMs, providing finer-grained visual features, significantly improving UI screen understanding and interaction capabilities, breaking existing method limitations.
Novelty
Ferret-UI is the first to apply 'any resolution' technology to UI screen understanding, combined with multimodal LLMs, significantly enhancing UI interaction precision and flexibility.
Limitations
- On complex UI screens, Ferret-UI may misidentify elements, especially when there are many widgets.
- The model may perform inconsistently across different operating systems.
Future Work
Future work could explore applying Ferret-UI to more operating systems and further optimizing the model's performance on complex UI screens.
AI Executive Summary
Ferret-UI is a multimodal LLM designed specifically for mobile UI screens, aiming to enhance UI screen understanding and interaction capabilities. Existing multimodal LLMs have limitations in handling UI screens, failing to effectively recognize and interact. Ferret-UI introduces 'any resolution' technology, dividing screens into sub-images to enhance visual features, improving the model's comprehension. Experimental results show that Ferret-UI surpasses GPT-4V in elementary UI tasks, particularly in icon recognition and text finding tasks. Additionally, in advanced tasks, Ferret-UI demonstrates excellent reasoning ability, especially in function inference tasks, outperforming Fuyu and CogAgent by 20%. However, the model may misidentify elements on complex UI screens. Future work could explore applying Ferret-UI to more operating systems and further optimizing the model's performance on complex UI screens.
Deep Analysis
Background
With the proliferation of mobile applications, UI screens have become crucial for user interaction. However, existing multimodal LLMs have limitations in handling UI screens, failing to effectively recognize and interact. Researchers have attempted to solve these issues by enhancing visual features and introducing new technologies.
Core Problem
Existing LLMs struggle with UI screens, failing to accurately recognize widgets and text, leading to inaccurate interactions. This is due to UI screens typically having elongated aspect ratios and smaller target objects.
Innovation
Ferret-UI introduces 'any resolution' technology, dividing screens into sub-images to enhance visual features. Combined with multimodal LLMs, it improves UI screen understanding and interaction capabilities.
Methodology
- �� Use 'any resolution' technology to divide screens into sub-images.
- �� Encode visual features using multimodal LLMs.
- �� Utilize training samples for instruction following and region annotation.
- �� Enhance the model's reasoning ability.
Experiments
Experiments used RICO and AMP datasets, covering elementary and advanced UI tasks. Ferret-UI's performance was evaluated against GPT-4V and other open-source models.
Results
Ferret-UI surpasses GPT-4V in elementary UI tasks, particularly in icon recognition and text finding tasks. Additionally, in advanced tasks, Ferret-UI demonstrates excellent reasoning ability.
Applications
Ferret-UI can be used for automated UI interaction and improving app accessibility, especially in scenarios requiring precise recognition and interaction.
Limitations & Outlook
The model may misidentify elements on complex UI screens, especially when there are many widgets. Additionally, the model may perform inconsistently across different operating systems.
Plain Language Accessible to non-experts
Imagine you're shopping in a large supermarket. Ferret-UI is like a super-smart shopping assistant that can quickly identify every product on the shelves and tell you their names and functions. Even if there are many products on the shelves, it can accurately find what you need and tell you how to use it. This is what Ferret-UI does in mobile apps; it quickly identifies every widget on the screen and helps users interact effectively.
ELI14 Explained like you're 14
Imagine you're playing a complex game with lots of buttons and icons. Ferret-UI is like a super assistant that helps you quickly identify each button's function and tells you how to use them. Even if the game interface is complex, it helps you find what you need, making your gameplay smoother. This is what Ferret-UI does in mobile apps; it helps you quickly identify every widget on the screen, making app usage easier.
Glossary
Multimodal Large Language Model (MLLM)
A language model combining multiple data modalities (e.g., text, images) to enhance understanding and interaction capabilities.
Used to enhance UI screen understanding and interaction.
Any Resolution
A technology that divides screens into sub-images to enhance visual features.
Used to improve UI screen visual features.
Visual Feature Encoding
Transforming visual information into features understandable by the model.
Used to enhance model understanding capabilities.
Instruction Following
The model's ability to perform operations based on instructions.
Used to improve UI interaction accuracy.
Region Annotation
Marking specific areas on the screen to enhance recognition capabilities.
Used to improve UI screen understanding.
Open Questions Unanswered questions from this research
- 1 How to improve the model's recognition ability on complex UI screens?
- 2 How to optimize model performance across different operating systems?
Applications
Immediate Applications
Automated UI Interaction
Enhance app interaction efficiency by quickly recognizing widgets on the screen.
Long-term Vision
Improving Accessibility
Help more users use apps through precise recognition and interaction.
Abstract
Recent advancements in multimodal large language models (MLLMs) have been noteworthy, yet, these general-domain MLLMs often fall short in their ability to comprehend and interact effectively with user interface (UI) screens. In this paper, we present Ferret-UI, a new MLLM tailored for enhanced understanding of mobile UI screens, equipped with referring, grounding, and reasoning capabilities. Given that UI screens typically exhibit a more elongated aspect ratio and contain smaller objects of interest (e.g., icons, texts) than natural images, we incorporate "any resolution" on top of Ferret to magnify details and leverage enhanced visual features. Specifically, each screen is divided into 2 sub-images based on the original aspect ratio (i.e., horizontal division for portrait screens and vertical division for landscape screens). Both sub-images are encoded separately before being sent to LLMs. We meticulously gather training samples from an extensive range of elementary UI tasks, such as icon recognition, find text, and widget listing. These samples are formatted for instruction-following with region annotations to facilitate precise referring and grounding. To augment the model's reasoning ability, we further compile a dataset for advanced tasks, including detailed description, perception/interaction conversations, and function inference. After training on the curated datasets, Ferret-UI exhibits outstanding comprehension of UI screens and the capability to execute open-ended instructions. For model evaluation, we establish a comprehensive benchmark encompassing all the aforementioned tasks. Ferret-UI excels not only beyond most open-source UI MLLMs, but also surpasses GPT-4V on all the elementary UI tasks.