GUI-CIDER: Mid-training GUI Agents via Causal Internalization and Density-aware Exemplar Reselection
GUI-CIDER enhances GUI agent training via causal internalization and density-aware exemplar reselection, boosting task success by 9.70%.
Key Findings
Methodology
GUI-CIDER employs a three-stage approach: data synthesis, exemplar reselection, and mid-training. In data synthesis, static planning and dynamic causal knowledge are extracted; exemplar reselection filters data by rewarding causal structures and penalizing semantic redundancy; mid-training embeds the refined data.
Key Results
- GUI-CIDER improves task success rates by an average of 9.70% on GUI knowledge and task completion benchmarks, reaching levels close to Claude-Sonnet-4.5 on 8B-scale agents.
- On the MMBench-GUI L1 benchmark, GUI-CIDER-8B significantly outperforms baselines, indicating enhanced GUI content understanding.
- Ablation studies show that removing the exemplar reselection stage significantly reduces SR, highlighting its importance.
Significance
GUI-CIDER significantly enhances the explicit internalization of world knowledge in GUI agents through mid-training, addressing inefficiencies in existing methods due to lack of explicit learning. This approach offers new insights for applying multimodal large language models in GUI operations.
Technical Contribution
GUI-CIDER's innovations in causal internalization and exemplar reselection significantly improve GUI agents' task completion and knowledge understanding without relying on external scaffolding, offering new engineering possibilities.
Novelty
GUI-CIDER is the first method to achieve mid-training of GUI agents through causal internalization and density-aware exemplar reselection, distinctly internalizing GUI world knowledge compared to traditional post-training methods.
Limitations
- Due to computational constraints, GUI-CIDER uses LoRA instead of full parameter tuning during mid-training, which may impact performance.
- Model parameters are limited to 4B to 8B, and effects on larger models remain unverified.
Future Work
Future research could explore the effectiveness under full parameter tuning and scale to larger models to further enhance GUI agents' knowledge internalization capabilities.
AI Executive Summary
Despite rapid advances in multimodal large language models for building GUI agents, their real-world task completion is bottlenecked by a lack of world knowledge about GUI operations. Existing solutions often rely on expensive multi-agent scaffolding or conventional post-training paradigms like supervised fine-tuning and reinforcement learning. However, these methods only allow agents to implicitly absorb world knowledge through action annotations or reward signals, leading to inefficient trajectory memorization rather than genuine comprehension.
To address this, we propose GUI-CIDER, a mid-training method that explicitly internalizes GUI world knowledge through causal internalization and density-aware exemplar reselection. GUI-CIDER operates in three stages: data synthesis extracts static planning and dynamic causal knowledge; exemplar reselection filters data by rewarding causal structures and penalizing semantic redundancy; mid-training embeds the refined data.
Extensive experiments on two GUI knowledge benchmarks and three task completion benchmarks demonstrate that GUI-CIDER consistently improves both the agent's understanding of GUI operations and its task success rates. The results suggest that knowledge scaling is a promising path toward more capable GUI agents.
Deep Dive
Abstract
Despite the rapid progress of multimodal large language models in building Graphical User Interface (GUI) agents, their real-world task completion is fundamentally bottlenecked by a lack of world knowledge about GUI operations. Existing solutions typically rely on expensive multi-agent scaffolding or conventional post-training paradigms, such as Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). However, post-training only allows agents to implicitly absorb world knowledge through action annotations or reward signals, leading to inefficient trajectory memorization rather than genuine comprehension. Therefore, an approach that enables explicit learning of this knowledge is imperative. To this end, we propose GUI-CIDER, a mid-training method that explicitly internalizes GUI world knowledge through Causal Internalization and Density-aware Exemplar Reselection. GUI-CIDER operates in three stages: (1) data synthesis, which distills static planning and dynamic causal knowledge from GUI trajectories into text; (2) exemplar reselection, which filters the corpus by rewarding causal structures and penalizing semantic redundancy; and (3) mid-training, where the refined data is used to embed the acquired knowledge. Extensive experiments on two GUI knowledge benchmarks and three task completion benchmarks demonstrate that GUI-CIDER consistently improves both the agent's understanding of GUI operations and its task success rates.The codes are available at https://github.com/Wuzheng02/GUI-CIDER.