SkillSelect-Serve: QoS-Aware Budgeted Skill Service Recommendation for LLM Agents
SkillSelect-Serve recommends QoS-aware budgeted skill services for LLM agents, improving hit rate to 0.9091.
Key Findings
Methodology
SkillSelect-Serve transforms raw skill documents into structured Skill Service Profiles and uses a lightweight requirement planner to convert tasks into structured requirement objects. It retrieves a high-recall candidate set via a shared discovery backbone and performs dual-granularity modeling at the service and bundle levels, selecting suitable service bundles through budget and QoS-constrained projection.
Key Results
- Evaluated on a registry of 35,353 skills, unconstrained recommendations fit only 9.1% of tasks, while constrained projection ensures 100% deliverability with only a 1.14-point hit rate loss.
- At an identical three-service budget, hit rate improves from 0.8864 (fixed Top-3 retrieval) to 0.9091.
- The constrained projection mechanism halves delivered risk exposure and eliminates 44-81% tool-violation rates of tool-agnostic recommendations.
Significance
This research redefines the management of reusable agent skills as discoverable, comparable, and constraint-aware service units, addressing challenges faced by small LLM agents in constrained environments. It significantly improves task deliverability and hit rate.
Technical Contribution
The method treats skills as service-oriented capability units, offering new theoretical guarantees and engineering possibilities. Unlike existing methods, it considers QoS constraints and budget limitations, providing a more efficient skill selection strategy.
Novelty
This is the first to model LLM agent skill selection as a budget-constrained skill service recommendation problem, offering a service-oriented perspective distinct from traditional document retrieval methods.
Limitations
- In complex tasks, there may be insufficient skill services to meet all requirements.
- Further research is needed on adjusting QoS constraints in dynamic environments.
Future Work
Future work could explore methods for real-time adjustment of QoS constraints in dynamic environments and further optimize the discovery and selection process of skill services.
AI Executive Summary
Reusable agent skills are emerging as a service-oriented capability layer for large language model (LLM) agents. SkillSelect-Serve addresses the challenge of selecting skill services for small LLM agents in constrained environments by transforming raw skill documents into structured skill services and using a lightweight requirement planner to convert tasks into structured requirement objects.
Evaluated on a registry of 35,353 skills, unconstrained recommendations fit only 9.1% of tasks, while constrained projection ensures 100% deliverability with only a 1.14-point hit rate loss. This mechanism also halves delivered risk exposure and eliminates 44-81% tool-violation rates of tool-agnostic recommendations.
This research redefines the management of reusable agent skills as discoverable, comparable, and constraint-aware service units, addressing challenges faced by small LLM agents in constrained environments. Future work could explore methods for real-time adjustment of QoS constraints in dynamic environments and further optimize the discovery and selection process of skill services.
Deep Analysis
Background
As large language model (LLM) agents evolve, agent skills, packaged as natural language instructions, task workflows, tool-use specifications, code templates, input-output assumptions, execution constraints, and risk warnings, become a critical intermediate layer between user tasks and executable capabilities. As skill libraries expand, selecting skills becomes more complex, especially for small LLM agents that can load only a few capability units.
Core Problem
Small LLM agents face challenges in selecting skill services in constrained environments. Existing fixed Top-k methods rank skills by textual relevance, overlooking requirement satisfaction, deliverability, and operational constraints.
Innovation
SkillSelect-Serve treats skills as service-oriented capability units, offering new theoretical guarantees and engineering possibilities. It considers QoS constraints and budget limitations, providing a more efficient skill selection strategy.
Methodology
- �� Transform raw skill documents into structured Skill Service Profiles
- �� Use a lightweight requirement planner to convert tasks into structured requirement objects
- �� Retrieve a high-recall candidate set via a shared discovery backbone
- �� Perform dual-granularity modeling at the service and bundle levels
- �� Select suitable service bundles through budget and QoS-constrained projection
Experiments
Evaluated on a registry of 35,353 skills using pooled multi-positive relevance judgments, the constrained projection ensures 100% deliverability with only a 1.14-point hit rate loss.
Results
The constrained projection mechanism halves delivered risk exposure and eliminates 44-81% tool-violation rates of tool-agnostic recommendations. At an identical three-service budget, hit rate improves from 0.8864 (fixed Top-3 retrieval) to 0.9091.
Applications
This method can be used by small LLM agents to select the most suitable skill services in constrained environments, improving task deliverability and hit rate.
Limitations & Outlook
In complex tasks, there may be insufficient skill services to meet all requirements. Further research is needed on adjusting QoS constraints in dynamic environments.
Plain Language Accessible to non-experts
Imagine you're in a restaurant with a menu of thousands of dishes, but you can only order a few. SkillSelect-Serve is like a smart ordering assistant that considers not only your taste preferences but also your budget, health conditions, and available utensils to recommend the most suitable dishes. It ensures that what you order is both delicious and meets your needs, rather than simply recommending the most popular items on the menu.
ELI14 Explained like you're 14
Imagine you're playing a game where your character has lots of skills, but you can only choose a few to defeat the enemy. SkillSelect-Serve is like a super helper that picks the best skill combo for you, making you unbeatable in the game! It considers not just the power of the skills but also your current gear and environment, ensuring you make the most of each skill. Isn't that cool?
Glossary
Skill Service
A skill service is a capability unit that packages natural language instructions, task workflows, and tool-use specifications.
In the paper, skill services are used as discoverable and comparable service units.
QoS (Quality of Service)
QoS refers to the quality attributes of a service in terms of functionality, cost, and risk.
The paper uses QoS to evaluate the suitability of skill services.
Token Budget
Token budget refers to the maximum number of tokens an agent can load.
Token budget is a crucial constraint when selecting skill services.
Risk Exposure
Risk exposure refers to the potential risks associated with using a service.
The paper reduces risk exposure to improve service safety.
Constrained Projection
Constrained projection refers to selecting service bundles under budget and QoS constraints.
Used to ensure the deliverability of skill services.
Open Questions Unanswered questions from this research
- 1 How to adjust QoS constraints in real-time to adapt to changing task requirements?
- 2 In complex tasks, how to ensure diversity and complementarity of skill services?
Applications
Immediate Applications
Small LLM Agent Optimization
Helps small LLM agents select the most suitable skill services in constrained environments, improving task deliverability and hit rate.
Long-term Vision
Intelligent Skill Management
Achieve smarter skill management and selection by dynamically adjusting QoS constraints to adapt to changing environments and demands.
Abstract
Reusable agent skills are emerging as a service-oriented capability layer for Large Language Model (LLM) agents. Unlike plain retrieval items, a skill exposes functional capabilities, input-output assumptions, tool dependencies, context cost, and risk metadata. Selecting skills is particularly challenging for small LLM agents, which can load only a few capability units under restricted context, tool availability, and risk tolerance. Existing fixed Top-k methods rank skills by textual relevance and overlook requirement satisfaction, deliverability, and operational constraints. We present SkillSelect-Serve, a QoS-aware, budget-constrained Skill Service recommendation framework. Raw skills are profiled as structured Skill Services, the task is converted into a structured requirement object, and candidates discovered from a large-scale registry are ranked by a calibrated task-conditioned suitability estimator and packed by a constrained projection enforcing token-budget, aggregated-risk, and tool-availability constraints, using only deployment-observable features. On a registry of 35,353 skills with pooled multi-positive relevance judgments verified by two independent assessors, the unconstrained top-5 recommendation fits a realistic 4,000-token context for only 9.1% of tasks; the constrained projection restores 100% deliverability at a cost of only 1.14 points of hit rate, outperforming retrieve-and-rerank, budget truncation, and diversity-based selection under identical budgets. The same mechanism halves delivered risk exposure and eliminates the 44-81% tool-violation rates of tool-agnostic recommendation. At an identical three-service budget, hit rate improves from 0.8864 to 0.9091 over fixed Top-3 retrieval. The results support managing reusable agent skills as discoverable, comparable, and constraint-aware service units instead of plain retrievable documents.