AutoTool: Dynamic Tool Selection and Integration for Agentic Reasoning

Recent progress in large language models (LLMs) has demonstrated impressive capabilities in reasoning, code generation, and multimodal understanding. Prior works like Toolformer, ReTool, and ToolGen have integrated static toolsets to enhance model performance, but these approaches assume fixed tools, limiting adaptability. As real-world applications demand models to operate in open-ended, evolving environments, the inability to dynamically select and incorporate unseen tools becomes a bottleneck. The challenge lies in enabling models to learn preferences over a large, diverse set of tools, including those not seen during training, while maintaining reasoning coherence and efficiency. This background underscores the necessity for a flexible, preference-driven tool selection mechanism that can generalize across tasks and environments.

The core problem is the static nature of existing tool integration methods, which restrict models to predefined toolsets. This leads to poor generalization when new tools emerge or environments change, severely limiting practical deployment. The difficulty is compounded by the need for models to not only select appropriate tools but also justify their choices, ensuring transparency and robustness. Addressing this requires a framework that supports dynamic, context-aware tool selection, capable of learning preferences over an evolving set of tools, and maintaining high reasoning accuracy across diverse tasks. The challenge is to design such a system that balances flexibility, interpretability, and computational efficiency.

This paper introduces AutoTool, a novel framework that models tool selection as a ranking problem, leveraging PL ranking and KL regularization to learn preferences over tools dynamically. Key innovations include:

1) Embedding-based tool representation, enabling generalization to unseen tools.

2) Dual-phase training: first stabilizing reasoning trajectories via supervised fine-tuning and RL, then refining tool preferences with PL ranking.

3) Explicit rationales for tool choices, improving interpretability.

4) Construction of a large, diverse dataset with annotated rationales, supporting preference learning.

These innovations collectively enable models to adaptively and accurately select tools in dynamic environments, surpassing prior static or fixed-tool methods.

• �� Assemble a large, diverse toolset (1000+ tools) and task set (100+ tasks) to create a comprehensive dataset.
• �� Map each tool into a shared embedding space using tool features, ensuring generalization.
• �� During trajectory generation, model generates a rationale and predicts an anchor embedding for tool selection.
• �� Sample tools based on embedding similarity, using a softmax over distances.
• �� Train in two phases: first, stabilize reasoning with supervised fine-tuning and RL; second, refine tool preferences via KL-regularized PL ranking.
• �� Formulate tool selection as a ranking problem, optimizing a distribution over candidate trajectories based on reward signals.
• �� Use a reward model evaluating both reasoning quality and final correctness to guide preference learning.
• �� Implement a loss function aligning the model’s policy distribution with the PL ranking distribution, ensuring preference consistency.

Experiments involve ten benchmarks across mathematics, science, search, and multimodal tasks, comparing AutoTool with models like GPT-4, Qwen, and ReTool. Metrics include accuracy and task success rate. The models are trained on datasets with explicit rationales, using different sizes (Qwen3-8B, Qwen2.5-VL-7B). Ablation studies assess the impact of PL ranking and data quality. The environment simulates evolving toolsets, testing the model’s ability to select unseen tools during inference. Results show consistent improvements, especially in zero-shot and unseen-tool scenarios, validating the effectiveness of the ranking-based preference learning approach.

AutoTool achieves an average of 6.4% higher accuracy across ten benchmarks, with notable gains in code generation (7.7%) and multimodal understanding (6.9%). The model demonstrates superior generalization, effectively leveraging unseen tools during inference. Ablation studies confirm that PL ranking and high-quality rationales are crucial for performance. The results highlight the importance of explicit preference modeling, enabling the model to prioritize high-quality tool sequences, thus reducing errors and improving reasoning coherence in complex, dynamic environments.

AutoTool can be applied in intelligent assistants, scientific research tools, and multimodal AI systems, where environments are constantly changing and new tools emerge. Its ability to dynamically select and justify tools enhances robustness and user trust. Long-term, this framework supports autonomous agents capable of continuous learning and adaptation, transforming AI from static systems into flexible, self-improving entities capable of operating in real-world, open-ended scenarios.

Despite promising results, the approach depends heavily on high-quality annotated rationales, which are costly to produce at scale. Its performance may decline in highly ambiguous or conflicting tool environments. Frequent tool invocation can impact inference speed, limiting real-time deployment. Additionally, the current framework may struggle with highly noisy or poorly represented tools, requiring further robustness improvements and efficiency optimizations.