RESOURCE2SKILL: Distilling Executable Agent Skills from Human-Created Multimodal Resources
RESOURCE2SKILL distills multimodal human resources into executable skills, improving agent performance by +11.9 points on average.
Key Findings
Methodology
The framework employs a hierarchical multimodal Skill Wiki, integrating tutorial videos, code, articles, and artifacts. Using the fθ model for feature extraction and AD for validation, it distills resources into structured skills. Offline, it builds a comprehensive skill repository; online, it retrieves and composes skills dynamically. Skills include text, executable code, visual examples, and metadata, ensuring multimodal complementarity. Experiments across seven domains show significant performance gains over no-skill baselines, with an average improvement of 11.9 percentage points and superior results in 26 of 28 model-domain pairs.
Key Results
- Across seven creative software domains, resource-distilled skills improved agent scores by an average of 11.9 points, with maximum gains of 30 points in Blender, Web, and UE5. The skill library reached 200 items, saturating performance. Online skill acquisition further boosted performance by 21.6 points in unseen tasks. Multimodal content and hierarchical retrieval contributed significantly to these improvements, demonstrating robustness and scalability.
Significance
This work addresses the limitations of traditional static, text-centric skill libraries by leveraging rich multimodal human-created resources, especially tutorial videos. It enables agents to acquire procedural and perceptual knowledge, bridging the gap between passive data and active execution. The approach enhances the autonomy and adaptability of software agents, fostering advances in intelligent automation and human-AI collaboration, with broad implications for industry and research.
Technical Contribution
The paper introduces a hierarchical multimodal Skill Wiki, combining vision-capable language models with resource validation components. It unifies offline knowledge distillation and online skill extension, supporting continuous learning. The integration of multiple content modalities—video, code, text—improves skill richness and retrieval efficiency. The system demonstrates superior performance across diverse domains, establishing a new paradigm for multimodal knowledge integration in AI agents.
Novelty
This is the first systematic effort to convert tutorial videos and other multimodal human resources into executable skills within a hierarchical wiki structure. Unlike prior work limited to text or code, this approach fuses multiple signals, enabling dynamic, scalable, and context-aware skill assembly. It addresses the challenge of heterogeneous data integration and real-time knowledge expansion, setting a new standard in multimodal skill learning.
Limitations
- The system heavily depends on high-quality multimodal resources; poor data quality or scarcity hampers performance. Generalization to highly novel or complex tasks remains limited. Computational costs for multimodal fusion and retrieval could hinder real-time deployment at scale. Future work should focus on resource quality control, efficient retrieval algorithms, and broader domain adaptation.
Future Work
Future directions include enhancing multimodal feature fusion efficiency, integrating reinforcement learning for optimal skill composition, and automating resource collection and annotation. Expanding to more complex, multi-task scenarios and improving generalization across diverse software environments are also key goals. Developing adaptive mechanisms for continual learning and knowledge updating will further advance autonomous, scalable AI agents.
AI Executive Summary
The evolution of intelligent software agents hinges on their ability to acquire and utilize procedural knowledge effectively. Traditional skill libraries, often handcrafted or solely text-based, fail to exploit the rich information embedded in tutorial videos and multimodal human resources. RESOURCE2SKILL presents a novel framework that bridges this gap by systematically distilling diverse multimodal content into a hierarchical, maintainable skill repository. This approach leverages vision-capable language models to extract procedural signals from videos, code snippets, and articles, organizing them into a structured Skill Wiki. Each skill entry combines textual explanations, executable code, visual exemplars, and metadata, enabling agents to retrieve and compose skills dynamically during task execution.
Experimental results across seven practical domains—such as slide design, web development, and 3D scene creation—demonstrate that agents equipped with RESOURCE2SKILL outperform baseline systems lacking skills by an average of 11.9 percentage points. The system's ability to scale the skill library to 200 items shows performance saturation, while online skill acquisition further enhances capabilities in unseen scenarios. Ablation studies confirm the importance of multimodal content, hierarchical organization, and source diversity.
This work signifies a major step toward autonomous, knowledge-rich AI agents capable of complex software operations. By integrating multimodal signals into a unified, scalable framework, RESOURCE2SKILL addresses longstanding limitations of static skill repositories. Its online extension mechanism ensures continuous learning, making it adaptable to evolving tasks and environments. The approach opens new avenues for research in multimodal learning, knowledge automation, and intelligent automation, promising broad industrial and academic impact.
Despite these advances, challenges remain in resource quality control, computational efficiency, and cross-domain generalization. Future efforts will focus on optimizing multimodal fusion, automating resource collection, and enhancing robustness. Overall, RESOURCE2SKILL sets a new benchmark for scalable, multimodal procedural knowledge in AI agents, paving the way for more autonomous and capable intelligent systems.
Deep Analysis
Background
软件代理的智能化发展经历了从规则驱动到深度学习的演变，早期依赖手工规则和静态知识库（如SkillFlow、Voyager），逐渐引入大规模预训练模型以提升理解能力。尽管如此，多模态资源（如视频教程、图像示例）在技能学习中的潜力尚未充分发挥，传统方法多依赖静态文本或代码，难以捕获操作的时序、视觉变化和设计风格。近年来，研究逐步关注多模态信息的融合与自动化知识提取，但仍面临异构数据整合、信息高维和动态扩展等挑战。本研究旨在突破现有局限，将多模态人类资源转化为可执行技能，推动软件代理在复杂任务中的自主性和泛化能力。
Core Problem
现有技能库多为静态、单一模态，难以充分利用教程视频等动态、多感知信息，导致技能表达不足、更新困难。如何自动提取多模态资源中的程序性和感知性信号，构建高效、可扩展的技能库，成为关键难题。特别是在复杂软件操作中，单一文本描述难以捕捉操作的时序、视觉效果和设计风格，限制了代理的自主学习和适应能力。
Innovation
提出层次化多模态技能维基，将视频、代码、文章融合，支持多源信息的互补表达。设计fθ和AD两个核心组件，实现资源的自动蒸馏和验证。采用统一的离线构建与在线扩展机制，支持持续知识更新。引入多模态内容的融合策略，提升技能的丰富性和检索效率。创新点在于将多模态信号转化为结构化、可执行的技能单元，突破了传统单一模态的限制。
Methodology
- �� 资源采集：从教程视频、代码仓库、文章和参考资料中获取多模态内容。• 蒸馏模型：利用vision-capable语言模型fθ提取关键帧、代码片段和文本信息，形成技能候选。• 验证机制：AD组件确保技能的完整性、可执行性和源可靠性。• 构建操作：离线蒸馏形成技能库，支持多模态内容的组织。• 推理阶段：根据用户需求，利用层次化索引检索相关技能，结合多模态信息进行组合。• 在线扩展：当离线库不足时，实时搜索新资源，蒸馏新技能，动态扩充库。
Experiments
在七个软件创作领域（如PPT、Web、Blender等）进行评估，使用多模态技能库与无技能对比。采用不同模型（GPT-5.4、GPT-5.5）和基线（ClaudeCode、Codex）进行性能对比。指标包括任务成功率、得分提升（平均11.9个百分点）、技能库规模影响和在线扩展效果。还进行了消融实验验证多源、多模态融合的必要性和层次化组织的优势。
Results
多模态技能显著提升代理性能，平均提升11.9个百分点，最大在UE5领域达40点。技能库规模达到200项时性能趋于饱和，在线扩展在新需求中提升21.6点。多源融合（视频、代码、文章）优于单一源，层次化索引显著减少检索时间。消融实验验证多模态内容和层次结构对性能的关键贡献，显示该方法在多软件场景中的普适性。
Applications
该技术可广泛应用于智能办公、自动化设计、虚拟仿真等领域，提升软件操作的智能化水平。通过构建丰富的技能库，代理能更好地理解复杂任务，减少人工干预。未来结合自主学习与自动资源采集，将实现更高效的知识更新和技能扩展，推动智能软件的普及。
Limitations & Outlook
当前方法依赖高质量多模态资源，资源不足或质量低劣时性能受限。模型在极端复杂或新颖任务中的泛化能力仍需提升。多模态融合和检索策略在大规模应用时可能面临计算成本和效率瓶颈。未来需优化资源筛选与知识更新机制，以增强系统的鲁棒性和扩展性。
Plain Language Accessible to non-experts
想象你在一家厨房做饭，里面有各种食材、工具和食谱。传统做法是看一本食谱书，按照步骤操作，但书里只告诉你文字描述，缺少直观的视觉和操作细节。现在，假设你可以看视频教程，看到厨师实际操作，学会了如何切菜、调味、摆盘。RESOURCE2SKILL就像是把这些视频、工具说明和食谱融合在一起，变成一套智能厨房助手。它能自动学习各种烹饪技巧，并在你需要时，快速找到合适的操作步骤，甚至自己学习新菜谱。这样，厨房变得更智能，做菜也更方便快捷。
ELI14 Explained like you're 14
想象你在学校学做手工艺品，老师会给你视频示范、图解步骤和一些样品。以前，你只能看文字说明，自己琢磨怎么做。而现在，有了这个新系统，就像有个超级助手，它把老师的视频、图片和说明都整理成一套聪明的指南。每次你想做新东西，只要告诉它你的想法，它就能帮你找到最合适的操作方法，还能自己学会新的技巧。这样一来，学手工变得更简单、更有趣，而且还能学到很多新东西，就像有个永远不会累的老师在帮你。
Glossary
Multimodal Resources (多模态资源)
包括视频、图像、文本和代码等多种信息形式，用于丰富知识表达。
在论文中指教程视频、文章、代码等多源信息。
Hierarchical Skill Wiki (层次化技能维基)
一种组织技能的结构化数据库，按领域和层级分类，支持检索和组合。
用于存储和管理提取的多模态技能。
fθ
基于视觉能力的语言模型，用于从多模态资源中提取关键信息。
实现资源蒸馏的核心模型。
AD (Validation Component)
技能验证机制，确保技能的完整性、可执行性和源可靠性。
保证技能质量的重要环节。
Resource-to-Skill (资源到技能)
将多模态人类资源自动转化为可执行技能的过程。
论文的核心创新。
Open Questions Unanswered questions from this research
- 1 如何进一步提升多模态资源的自动筛选与质量控制，确保在低质量资源环境下的性能稳定。
- 2 多模态融合的效率优化，尤其在大规模应用中的计算成本与实时性问题。
- 3 跨领域迁移能力不足，如何让技能库在不同软件或任务间快速适应。
Applications
Immediate Applications
智能办公助手
利用多模态技能库，自动生成PPT、网页、表格等，提高办公效率，减少人工操作。
自动化设计工具
在建筑、动画等行业中，快速构建场景和模型，降低专业门槛，提升设计速度。
Long-term Vision
自主学习与知识更新
未来系统能自主从网络资源中学习新技能，持续扩展知识库，实现真正的自主智能。
Abstract
Skills are a useful abstraction for software agents, turning human and agent experience into reusable procedural knowledge. Yet existing skill libraries are mostly hand-written, text-centric, or derived from agent traces, leaving tutorial videos and other multimodal human resources largely underused. We present RESOURCE2SKILL, a framework that distills multimodal resources, including tutorial videos, repositories, articles, and reference artifacts, into executable skills for software agents. RESOURCE2SKILL organizes these skills as a hierarchical multimodal Skill Wiki, where each entry combines structured text, code, visual examples, metadata, and provenance. This design preserves complementary signals from different resources: videos capture temporal operations and visual effects, code captures executable tool patterns, and articles or artifacts provide conceptual and stylistic grounding. At inference time, agents retrieve and compose relevant skills from the wiki; when coverage is insufficient, the same construction operator can acquire new skills online. Across seven practical authoring domains, RESOURCE2SKILL improves average overall score by +11.9 percentage points over no-skill agents and outperforms strong harness baselines in 26 of 28 main-aggregate model-domain cells. Ablations confirm the value of multimodal skill format, hierarchical organization, source diversity, selection strategy, and online acquisition.