AUSO: Action-Level Unified Skill Optimization from Internalization to Utilization

The evolution of large language models (LLMs) has transformed natural language processing, enabling models like GPT-4 and PaLM to perform complex reasoning and decision-making tasks. Early work focused on chain-of-thought prompting and external tool integration, such as API calls and web navigation, to enhance capabilities. Reinforcement learning from human feedback (RLHF) and agentic RL further pushed the boundary, allowing models to interact with environments and optimize behavior based on delayed rewards. However, these methods often treat procedural knowledge—skills—as external modules or static parameters, limiting flexibility and transferability. Recent efforts like Skill0.5 attempted hybrid approaches, internalizing general skills while maintaining explicit task-specific skills, but still relied on coarse trajectory success signals for skill routing. This coarse granularity hampers precise credit assignment at the action level, especially in complex, multi-step tasks, leading to suboptimal skill utilization and poor generalization. The need for a more fine-grained, adaptive approach to skill management has become evident, motivating the development of AUSO.

Existing skill learning frameworks predominantly use trajectory-level success rates to determine when to internalize or utilize skills, resulting in fuzzy decision boundaries. This coarse granularity prevents the model from accurately assessing the contribution of individual actions, especially in complex environments with sparse rewards. Consequently, the model struggles to dynamically adapt skill use at the action level, limiting its ability to transfer skills across tasks and environments. The challenge lies in developing a mechanism that can evaluate the impact of skills on a per-action basis, enabling more precise credit assignment and flexible skill deployment. Addressing this gap is critical for improving the robustness, efficiency, and generalization of autonomous decision-making systems.

AUSO introduces a novel action-aware optimization framework that leverages Jensen-Shannon divergence (JSD) to quantify how skill guidance influences each action’s distribution. This enables a seamless transition from external skill supervision to autonomous, context-sensitive skill utilization. The key innovations include:

• �� Action-level skill sensitivity measurement via JSD, capturing the discrepancy between skill-conditioned and skill-free policies.
• �� Multi-stage training schedule: initial teacher-guided internalization, followed by autonomous exploration, and finally action-level evaluation.
• �� Dynamic reweighting of policy gradients based on action-specific information gain and trajectory advantage, allowing the model to focus on beneficial skills.
• �� A unified learning paradigm that replaces coarse trajectory routing with fine-grained, action-level assessment, improving transferability and robustness.

• �� Stage 1: Teacher-guided skill internalization—using a teacher conditioned on general skills to guide the student policy via JSD-based discrepancy measurement.
• �� Calculate the divergence between teacher and student action distributions at each step, normalize, and convert into an action weight.
• �� Stage 2: Autonomous exploration—coupling the action-level skill sensitivity signal with trajectory advantage to reweight policy updates, reinforcing actions benefiting from skills.
• �� Stage 3: Skill evaluation and utilization—assessing each sampled action under both skill-conditioned and skill-free contexts, dynamically adjusting skill use based on action-level benefit.
• �� The training schedule employs a ramp-up and decay function for supervision strength, ensuring a smooth transition from external guidance to autonomous, adaptive skill use.

The evaluation involved ALFWorld, WebShop, and SearchQA datasets, comparing AUSO against baselines like Skill0.5 and SkillRL. Metrics included success rate, generalization to out-of-distribution tasks, and multi-hop reasoning accuracy. Hyperparameters such as the JSD coefficient, training stages, and reward discounting were tuned via ablation studies. Results showed consistent improvements: success rates increased by 6-8%, especially in OOD scenarios. The experiments also included ablation tests confirming the importance of the JSD-based action signal, and analysis of the training schedule’s impact on performance stability.

AUSO outperformed baseline methods across all datasets, with success rate improvements of up to 8% in ALFWorld and 6% in WebShop. It demonstrated superior generalization in OOD tasks, maintaining high success rates where baselines declined. The ablation studies highlighted that removing the JSD-based action signal reduced performance by 4-5%, confirming its critical role. The multi-stage training schedule was shown to facilitate better skill internalization and adaptive utilization, especially in complex multi-hop reasoning tasks in SearchQA. Overall, AUSO’s fine-grained, action-level approach significantly enhanced decision accuracy and transferability.

This framework is applicable to autonomous robots, virtual assistants, and decision-making systems requiring adaptive skill management. It enables agents to better transfer learned skills across diverse tasks and environments, improving robustness and efficiency. The approach can be integrated into existing RL pipelines, especially in scenarios with sparse rewards and complex action spaces. Long-term, AUSO could facilitate the development of general-purpose autonomous agents capable of continuous learning and adaptation in real-world settings, such as household robotics or industrial automation.

The reliance on environment feedback and teacher guidance may limit applicability in real-world scenarios with sparse or noisy signals. Computational overhead from calculating JSD at each action step poses scalability challenges, especially in high-dimensional or real-time applications. The multi-stage training process adds complexity and hyperparameter sensitivity, requiring careful tuning. Future work should focus on reducing computational costs, improving robustness in sparse feedback environments, and extending the framework to multi-agent and real-world robotic systems.