K^2-Agent: Co-Evolving Know-What and Know-How for Hierarchical Mobile Device Control
K²-Agent enhances mobile device control via co-evolving 'know-what' and 'know-how', achieving a 76.1% success rate.
Key Findings
Methodology
K²-Agent employs a hierarchical framework, dividing tasks into 'know-what' and 'know-how'. The high-level planner evolves through an SRLR loop, while the low-level executor is trained using the C-GRPO algorithm, forming a closed-loop system. The high-level extracts task knowledge from a single demonstration, and the low-level acquires skills through dynamic demonstration injection and error-decoupled replay balancing.
Key Results
- On the AndroidWorld benchmark, K²-Agent achieved a 76.1% success rate, surpassing existing open-source models and demonstrating strong capabilities in complex tasks.
- In ScreenSpot-v2 and AitW, K²-Agent's low-level executor excelled in unseen tasks, validating its broad applicability.
- Ablation studies confirmed the critical role of the SRLR loop and C-GRPO algorithm in performance improvement.
Significance
K²-Agent addresses the challenges of long-horizon planning and precise operation in mobile device control by applying human cognitive models, advancing research in this field. Its dual generalization capability allows it to perform well across different models and tasks, offering broad application potential.
Technical Contribution
K²-Agent introduces SRLR loops and C-GRPO algorithms, achieving separate evolution of task knowledge and skills, providing new theoretical guarantees and engineering possibilities, with significant advantages over existing methods.
Novelty
K²-Agent is the first to apply the separation of 'know-what' and 'know-how' from human cognition to mobile device control, offering a fundamentally new framework for task planning and execution compared to existing methods.
Limitations
- In some complex tasks, K²-Agent may require more demonstrations to improve success rates.
- Its performance may be limited by hardware constraints under extreme conditions.
Future Work
Future research directions include extending K²-Agent to more application scenarios, optimizing its performance on different hardware, and exploring more task types.
AI Executive Summary
Existing mobile device control agents often struggle with complex tasks due to a lack of relevant task experience or unfamiliarity with skill execution. K²-Agent offers a hierarchical framework that separates and co-evolves 'know-what' and 'know-how' knowledge, modeling human cognition. The high-level planner extracts and iteratively refines task-level knowledge through an SRLR loop from a single demonstration, while the low-level executor is trained using C-GRPO, constructing a balanced sample pool and employing dynamic demonstration injection to guide successful trajectory generation. On the AndroidWorld benchmark, K²-Agent achieves a 76.1% success rate using only raw screenshots and open-source backbones. Furthermore, K²-Agent demonstrates powerful dual generalization: its high-level knowledge transfers across diverse models, while its low-level skills perform well on unseen tasks.
Deep Analysis
Background
The field of mobile device control has long faced challenges in long-horizon planning and precise operation. Traditional methods often rely on extensive manual design or massive amounts of data and computational resources, struggling to achieve ideal results in complex tasks. Recently, more research has begun to separate reasoning from action or adopt explicit planner-executor hierarchies.
Core Problem
Existing mobile device control methods perform poorly on complex tasks requiring long-horizon planning and precise operations. This is due to a lack of relevant task experience or unfamiliarity with skill execution, leading to poor task generalization.
Innovation
K²-Agent's core innovation lies in its hierarchical framework, dividing tasks into 'know-what' and 'know-how'. By implementing SRLR loops and C-GRPO algorithms, it achieves separate evolution of task knowledge and skills, offering a new framework for task planning and execution.
Methodology
- �� The high-level planner extracts task knowledge through an SRLR loop. • The low-level executor acquires skills via the C-GRPO algorithm. • The two form a closed-loop system connected by sub-goals. • Dynamic demonstration injection and error-decoupled replay balancing enhance training efficiency.
Experiments
On the AndroidWorld benchmark, K²-Agent achieved a 76.1% success rate using only raw screenshots and open-source backbones. The experimental design includes tasks of varying difficulty, with comparisons against multiple baseline methods.
Results
K²-Agent achieved a 76.1% success rate on the AndroidWorld benchmark, surpassing existing open-source models. Its performance in ScreenSpot-v2 and AitW validates its skills' broad applicability.
Applications
K²-Agent can be used for automating mobile device operations, especially in complex tasks requiring long-horizon planning and precise operations. Its dual generalization capability allows it to perform well across different models and tasks.
Limitations & Outlook
K²-Agent may require more demonstrations to improve success rates in some complex tasks. Its performance may be limited by hardware constraints under extreme conditions. Future research directions include extending its application scenarios and optimizing performance.
Plain Language Accessible to non-experts
Imagine a chef preparing a complex dinner in a kitchen. The chef needs to know the recipe for each dish ('know-what') and how to cook them ('know-how'). K²-Agent is like this chef, completing complex tasks by separating and co-evolving these two types of knowledge. The high-level planner is like the chef's brain, planning the steps for each dish. The low-level executor is like the chef's hands, mastering each step through practice.
ELI14 Explained like you're 14
Imagine you're playing a mobile game with multiple tasks. You need to know the goal of each task ('know-what') and how to operate the phone to complete them ('know-how'). K²-Agent is like your game assistant, helping you understand task goals and guiding you on how to operate. It learns and practices to improve its ability to complete tasks.
Glossary
K²-Agent
A hierarchical framework for mobile device control that separates and co-evolves 'know-what' and 'know-how' knowledge.
K²-Agent implements task planning and execution through SRLR loops and C-GRPO algorithms.
SRLR Loop
A self-evolution loop consisting of Summarize, Reflect, Locate, and Revise stages.
Used by the high-level planner to extract and optimize task knowledge.
C-GRPO Algorithm
An algorithm for training the low-level executor, enhancing training efficiency through error-decoupled replay balancing and dynamic demonstration injection.
Used by the low-level executor to acquire skills.
AndroidWorld
A benchmark for evaluating mobile device control methods, consisting of 116 tasks.
K²-Agent achieved a 76.1% success rate on this benchmark.
Dual Generalization
K²-Agent's high-level knowledge and low-level skills perform well across different models and tasks.
Validates K²-Agent's broad applicability.
Open Questions Unanswered questions from this research
- 1 How to improve K²-Agent's performance under extreme conditions?
- 2 Can K²-Agent be applied to more task types?
Applications
Immediate Applications
Automated Mobile Device Operations
K²-Agent can automate complex mobile device operations, especially in tasks requiring long-horizon planning.
Long-term Vision
Intelligent Assistant
K²-Agent can serve as an intelligent assistant, helping users complete various complex mobile device tasks.
Abstract
Existing mobile device control agents often perform poorly when solving complex tasks requiring long-horizon planning and precise operations, typically due to a lack of relevant task experience or unfamiliarity with skill execution. We propose K2-Agent, a hierarchical framework that models human-like cognition by separating and co-evolving declarative (knowing what) and procedural (knowing how) knowledge for planning and execution. K2-Agent's high level reasoner is bootstrapped from a single demonstration per task and runs a Summarize-Reflect-Locate-Revise (SRLR) loop to distill and iteratively refine task-level declarative knowledge through self-evolution. The low-level executor is trained with our curriculum-guided Group Relative Policy Optimization (C-GRPO), which (i) constructs a balanced sample pool using decoupled reward signals and (ii) employs dynamic demonstration injection to guide the model in autonomously generating successful trajectories for training. On the challenging AndroidWorld benchmark, K2-Agent achieves a 76.1% success rate using only raw screenshots and open-source backbones. Furthermore, K2-Agent shows powerful dual generalization: its high-level declarative knowledge transfers across diverse base models, while its low-level procedural skills achieve competitive performance on unseen tasks in ScreenSpot-v2 and Android-in-the-Wild (AitW).