SkillLens: Visual Skill Cards for Retrieval-Augmented GUI Action Prediction and On-Policy Distillation
SkillLens uses Visual Skill Cards to improve GUI action prediction, boosting Step SR+11.6.
Key Findings
Methodology
This paper introduces Visual Skill Cards (VSCs) as a state-conditioned memory representation, combining reusable procedures, applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs via Trace-to-VSC, converting heterogeneous interaction data into a unified schema. During inference, relevant VSCs are retrieved and selectively expanded to load only necessary evidence, conditioning a fixed visual-language model executor for grounded GUI action prediction. The system separates retrieval from evidence expansion, balancing runtime efficiency with visual detail. Hierarchical organization of skills further supports multi-source data integration, enabling effective generalization across tasks.
Key Results
- On Mind2Web and WebLINX-BrowserGym benchmarks, SkillLens improved gpt-4o's Step SR by +11.6 points and overall performance by +2.9, demonstrating robustness in complex GUI tasks.
- CardDistill further transferred VSC-based knowledge into student models, achieving +12.0 Step SR and +3.2 overall improvements, validating the effectiveness of knowledge distillation.
- Ablation studies confirmed that selective evidence loading, hierarchical skills, and multi-source data fusion significantly enhance accuracy and efficiency, confirming the design's validity.
Significance
This work addresses the longstanding challenge of procedural visual memory in GUI agents. By externalizing structured visual procedural knowledge through VSCs, models can better recognize workflows, disambiguate similar controls, and verify progress, leading to more reliable and generalizable automation. The combination of retrieval and distillation frameworks paves the way for scalable, interpretable, and adaptable AI systems in software automation, virtual assistants, and beyond. It bridges the gap between raw interaction data and practical deployment, offering a new paradigm for multi-modal knowledge management in AI systems.
Technical Contribution
The core technical contribution is the design of VSCs as a unified, structured memory module that encapsulates procedures, cues, and visual evidence. The Trace-to-VSC pipeline standardizes heterogeneous data, enabling cross-source integration. The retrieval-expansion mechanism efficiently balances evidence quality and computational cost, while the hierarchical organization supports long-horizon tasks. CardDistill innovatively uses VSCs as privileged teacher signals during training, facilitating knowledge transfer without runtime retrieval. The framework integrates advanced retrieval algorithms, fixed visual-language models, and distillation techniques, forming a comprehensive solution for GUI understanding.
Novelty
This is the first work to formalize Visual Skill Cards as a structured, reusable, and hierarchical memory unit for GUI tasks, bridging the gap between raw interaction traces and parameterized skills. Unlike prior approaches relying solely on text or long demonstrations, VSCs encode visual and procedural information compactly, enabling efficient retrieval and grounding. The combination of Trace-to-VSC, selective evidence expansion, and source-aligned distillation represents a novel architecture that advances the state-of-the-art in visual procedural memory and grounded action prediction.
Limitations
- The approach depends heavily on high-quality interaction data and accurate source adapters; poor data quality can impair VSC effectiveness.
- In highly dynamic or novel interface environments, VSCs may require frequent updates, posing challenges for real-time adaptation.
- Handling rare or unseen workflows remains difficult, necessitating online learning or continual updating mechanisms to improve coverage.
Future Work
Future research will focus on automating VSC updates through online learning, improving scalability across diverse platforms, and integrating reinforcement learning for adaptive retrieval strategies. Exploring online knowledge base maintenance and continual learning will enhance robustness in real-world applications. Extending VSCs to multi-modal inputs and more complex workflows will further broaden their applicability, ultimately enabling autonomous agents capable of seamless multi-platform operations.
AI Executive Summary
SkillLens introduces Visual Skill Cards (VSCs) as a structured external memory to enhance GUI action prediction. Traditional models rely on raw interaction traces or text-only skills, which lack the ability to encode visual procedural context. VSCs encapsulate workflows, applicability cues, visual evidence, and verification signals into compact, reusable units. The system constructs these cards via Trace-to-VSC, converting heterogeneous data sources into a unified schema. During inference, relevant VSCs are retrieved using a lightweight index, and only essential evidence is selectively loaded, reducing runtime costs while preserving visual detail.
This approach separates retrieval from evidence expansion, enabling efficient grounding of actions in real-time interface states. Hierarchical organization of skills supports long-horizon tasks, while the fixed visual-language executor predicts actions conditioned on live interface and evidence. The training phase employs CardDistill, which leverages VSCs as privileged teacher signals to transfer knowledge into student models, eliminating the need for runtime retrieval.
Experimental results on benchmarks like Mind2Web and WebLINX-BrowserGym show that SkillLens boosts performance metrics significantly, with +11.6 Step SR improvement over baseline gpt-4o. The distilled models further gain +12.0 in Step SR, demonstrating effective knowledge transfer. Ablation studies confirm the importance of selective evidence loading and hierarchical skills.
Overall, this work advances GUI automation by providing a scalable, interpretable, and efficient framework for visual procedural memory. It opens avenues for robust, generalizable AI agents capable of complex multi-modal interactions, with promising applications in software automation, virtual assistants, and industrial automation. Future directions include dynamic VSC updating, online learning, and broader multi-platform deployment, aiming to realize fully autonomous, adaptable GUI agents.
Deep Analysis
Background
随着人工智能在软件自动化中的快速发展，基于视觉-语言模型（VLMs）的GUI理解逐渐成为研究焦点。早期方法多依赖规则或长轨迹示范，难以实现高效泛化。近年来，OpenAI的GPT系列、Google的Gemini等模型在多模态理解方面取得突破，但在GUI任务中仍面临程序记忆不足的问题。现有的技能包和知识库多为参数化流程或长轨迹，缺少结构化、可检索的外部记忆，限制了模型在复杂场景中的表现。视觉记忆与检索技术虽有所发展，但多集中于文档或对话历史，缺少针对GUI状态的程序化封装。SkillLens的创新在于提出VSC，将流程、线索与视觉证据封装为结构化单元，为GUI任务提供了新的知识管理路径。
Core Problem
当前GUI动作预测模型多依赖静态的训练数据或长轨迹示范，难以在实时环境中高效利用外部知识。缺乏结构化的视觉程序记忆导致模型在识别相似控件或确认流程状态时易出错。复杂界面中的相似元素难以区分，模型常因缺乏上下文感知而误操作。如何在保持模型架构不变的情况下，动态引入丰富的视觉流程知识，成为关键难题。解决方案需兼顾运行效率、知识的可重用性与可解释性，满足工业应用对高效、可靠的GUI理解需求。
Innovation
本研究的核心创新在于提出Visual Skill Cards（VSCs）作为一种状态条件的外部记忆，将流程、适用性线索、视觉证据与验证信号封装为结构化单元。Trace-to-VSC机制实现了异构交互数据的标准化，支持多源数据的封装与层级组织。SkillLens通过检索-扩展机制，选择性加载关键证据，避免全量加载带来的成本。引入层级化技能组织，支持长短期任务的封装与调用。CardDistill利用VSC作为教师信号，将知识迁移到学生模型，实现无检索推理。这一框架突破了传统的纯文本或长轨迹示范的局限，为GUI理解提供了高效、结构化的知识封装方案。
Methodology
- �� 将异构交互数据（网页、桌面操作记录）转化为标准化的VSC格式，封装流程、线索、视觉证据与验证信息。
- �� 通过Trace-to-VSC机制，将轨迹片段总结为可重用的技能卡，支持层级化组织。
- �� 在推理时，利用索引机制（如文本索引或VLM-board）检索相关VSCs，筛选出最匹配的候选集。
- �� 采用局部证据扩展策略，只加载所选VSC的高分辨率视图，结合实时界面状态进行动作预测。
- �� 设计固定的视觉-语言模型执行器，条件于加载的证据与界面，输出 grounded GUI动作。
- �� 训练阶段，利用VSC作为教师信号，通过CardDistill将知识迁移到学生模型，实现无检索推理。
- �� 实验中，评估模型在Mind2Web、WebLINX-BrowserGym和OSWorld-G上的性能变化，验证VSC的有效性与蒸馏的迁移效果。
Experiments
采用多场景基准（Mind2Web、WebLINX-BG、OSWorld-G），比较不同模型（如gpt-4o、Qwen3-VL-2B、Gemini 2.5）在Step SR、元素识别、整体性能等指标上的提升。设置不同VSC条件（全技能、无技能、只加载关键证据）进行消融分析，验证VSC的贡献。采用文本索引、VLM-board等检索策略，评估运行成本与性能折中。训练中，利用CardDistill将VSC知识迁移到学生模型，观察其在无检索条件下的表现变化。多指标、多场景的对比验证了VSC机制的有效性。
Results
VSC机制在Mind2Web中使Step SR提升11.6点，整体性能提升2.9，优于无技能基线。在WebLINX-BG上，Step SR+3.2，性能持续改善。引入CardDistill后，学生模型在两个基准上分别达到了Step SR+12.0和Overall+3.2，验证了知识迁移的有效性。消融实验显示，关键证据加载和层级化技能对性能提升至关重要。不同检索策略（如VLM-board）在不同场景表现差异，验证了机制的灵活性。整体来看，VSC不仅提升了准确率，也降低了推理成本，展现出良好的实用潜力。
Applications
该技术适用于自动化软件操作、智能助手、企业流程自动化等场景，尤其在复杂界面理解与操作中表现优越。只需提供界面截图和任务描述，模型即可自主识别控件、确认流程状态，减少人工调试。未来结合在线学习和持续更新机制，可实现动态知识库维护，适应多变的应用环境。长远来看，VSC框架有望推动多模态交互系统的智能升级，实现更高效、更可靠的自动化操作。
Limitations & Outlook
目前方法依赖高质量交互数据和源适配器，数据偏差可能影响泛化能力。在极端复杂或动态界面中，VSC的适应性和更新机制仍需优化。此外，模型在处理未覆盖的长尾场景或新流程时表现有限，需结合在线学习策略。证据加载仍存在一定成本，未来需优化检索与加载效率。整体而言，未来需解决知识更新、场景适应和成本控制等关键问题。
Plain Language Accessible to non-experts
想象你在一家厨房里做饭。每次做菜前，你会准备好食材、步骤和工具的清单。SkillLens就像是厨房里的智能助手，它记住各种菜谱（流程），知道什么时候用哪些食材（界面元素），还会根据你做菜的状态（界面变化）提醒你下一步。它会把这些信息整理成一张“菜谱卡”，每次你想做菜时，只需找出对应的卡片，助手就会告诉你需要准备的食材、操作步骤和确认菜是否做好。这样，无论你做什么菜，都能快速找到正确的步骤，不会迷路，也不用每次都重新查资料。它让做饭变得简单又高效，甚至还能学会新菜谱，变成厨房里的超级助手。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，比如冒险游戏。每次你遇到新任务，都需要记住怎么操作，比如怎么找到宝藏、打败怪物或者解谜。SkillLens就像你的游戏助手，它会把每个任务的关键步骤和提示都写成一张“技能卡”。当你再次遇到类似的任务时，只要查找相关的卡片，它就会告诉你下一步该做什么，还会帮你确认你是否做对了。这样，你不用每次都从头学，也不用记一大堆细节，只要看一眼技能卡，就知道怎么继续。它让游戏变得更容易，也让你变成更厉害的玩家！
Glossary
Visual Skill Card (VSC, 视觉技能卡)
一种封装流程、线索和视觉证据的结构化记忆单元，用于GUI动作预测。它结合操作流程、适用性线索、界面视觉信息和验证信号。
在本文中，VSC作为外部知识存储，用于增强模型的程序记忆和推理能力。
Trace-to-VSC (轨迹转技能卡)
将异构交互数据转化为标准化的技能卡的机制，提取可重用的流程片段和视觉证据。
用于构建VSC库，实现多源数据的封装与层级组织。
Grounded GUI Action (基础GUI动作)
基于实时界面状态和证据，预测并执行的具体用户操作（如点击、输入等）。
模型在推理时，依赖VSC和实时界面信息进行 grounded 动作预测。
CardDistill (技能卡蒸馏)
利用VSC作为教师信号，将知识迁移到学生模型，实现无检索的推理能力。
训练阶段，将VSC知识融入模型参数，减少推理成本。
Open Questions Unanswered questions from this research
- 1 如何自动扩展VSC库以适应不断变化的界面和任务场景，仍需研究动态更新机制。未来需要探索结合在线学习和持续训练的方法，以保持知识的时效性和适应性。
Applications
Immediate Applications
自动化软件操作
利用SkillLens实现自动化测试、数据录入和界面导航，减少人工干预，提高效率。只需提供界面截图和任务描述，模型即可自主识别控件并执行操作。
智能助手
集成到桌面或网页助手中，帮助用户完成复杂操作，提升用户体验。模型能理解界面状态，提供实时操作建议。
Long-term Vision
跨平台智能交互系统
将VSC技术推广到不同操作系统和应用场景，实现跨平台的自动化和智能化操作，推动工业自动化和智能办公的普及。
Abstract
Computer-using agents can perceive rich software interfaces, yet their decisions often lack visual procedural memory: they may recognize individual controls without identifying which familiar workflow is active, which control matters next, or what evidence would confirm progress. Raw interaction traces preserve such information but are long and noisy to condition on, whereas text-only skills often omit the visual state that makes a procedure applicable. We introduce Visual Skill Cards (VSCs), a state-conditioned memory representation that binds reusable procedures with applicability cues, visual evidence, and verification signals. SkillLens constructs VSCs from heterogeneous interaction experience through Trace-to-Visual-Skill-Card and, at inference time, retrieves relevant cards and selectively expands only the evidence needed by a fixed visual-language model executor for grounded GUI action prediction. The same representation also supports CardDistill, which uses VSC evidence as privileged teacher context to train a student that acts without runtime card retrieval. Across Multimodal-Mind2Web and WebLINX-BrowserGym, SkillLens improves the frozen GPT-5.4-mini executor by +11.6 points in Step SR and +2.9 points in Overall, respectively; CardDistill further improves the corresponding student-only Qwen3-VL-2B metrics by +12.0 and +3.2 points.