CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models
CREATOR enables LLMs to generate tools via documentation and code, disentangling abstract creation from concrete reasoning, improving performance on math and tabular tasks.
Key Findings
Methodology
CREATOR comprises four stages: creation, decision, execution, and rectification. It leverages LLMs’ abstract reasoning to generate generalizable tools with documentation and code, avoiding API reliance. The framework separates tool creation from decision-making, enhancing robustness. Prompts and demonstrations guide models, with error traceback enabling automatic correction. Experiments on MATH and TabMWP show superior accuracy over chain-of-thought, program-of-thought, and tool-using baselines.
Key Results
- On the MATH dataset, CREATOR achieves 59.7% accuracy, outperforming chain-of-thought (37.9%) and program-of-thought (46.5%). On TabMWP, it reaches 94.7%. On the new Creation Challenge dataset, it attains 75.5% average accuracy. The ability to create tools significantly improves generalization, especially in unseen or complex problems.
- Disentangling creation and decision stages facilitates knowledge transfer across similar questions. Error traceback and automatic correction boost accuracy by about 10%. Providing hints further enhances tool creation efficiency, demonstrating the framework’s robustness.
- Core innovation lies in integrating tool creation into the reasoning process, supporting non-linear, multi-stage reasoning paths. This approach surpasses traditional linear methods, enabling models to autonomously generate and refine tools, thus solving more complex problems effectively.
Significance
This work advances the field by shifting from static tool use to dynamic tool creation, addressing key limitations in model reasoning stability and adaptability. It enhances the ability of LLMs to handle complex, multi-step tasks in mathematics and data analysis, paving the way for more autonomous AI systems. The framework’s robustness and flexibility could revolutionize AI problem-solving, making models more self-reliant and capable of continuous learning and adaptation.
Technical Contribution
Technically, CREATOR introduces a four-stage process that combines abstract reasoning, code-based tool creation, and automatic error correction. It departs from prior methods by enabling models to generate diverse, reusable tools rather than relying solely on predefined APIs. The integration of code as a medium supports complex, multi-step reasoning with error tracebacks for self-correction. This design enhances robustness and generalization, setting a new standard for autonomous tool generation in LLMs.
Novelty
This is the first comprehensive framework that enables large models to autonomously create, modify, and utilize tools during reasoning. Unlike existing approaches limited to tool invocation, CREATOR supports dynamic tool generation based on problem characteristics, with a non-linear, multi-stage reasoning process. Its automatic correction mechanism and focus on generalizability distinguish it from prior work, marking a significant step forward in AI autonomy.
Limitations
- The framework relies heavily on demonstrations and prompts, which may limit scalability and generalization in zero-shot or unseen scenarios. Further research is needed to reduce dependency on extensive exemplars.
- Error correction based on traceback may struggle with complex, multi-error situations, requiring more sophisticated mechanisms.
- Computational costs increase due to multiple reasoning and correction cycles, which could hinder real-time applications, especially with larger models.
Future Work
Future directions include reducing dependence on demonstrations, enhancing automatic correction mechanisms, and expanding tool types to include multi-modal functionalities. Integrating reinforcement learning to improve tool creation quality, and applying the framework to broader domains like scientific discovery or industrial automation, are promising avenues. Additionally, optimizing computational efficiency and exploring unsupervised or semi-supervised training could further advance autonomous tool creation.
AI Executive Summary
CREATOR introduces a groundbreaking framework for large language models, empowering them to autonomously generate and refine tools through a four-stage process: creation, decision, execution, and rectification. Traditional approaches primarily relied on predefined APIs or linear reasoning, which limited flexibility and robustness, especially in complex or unseen problems. CREATOR addresses these limitations by enabling models to abstractly create versatile tools, represented as code, tailored to specific tasks. This separation of tool creation from decision-making allows for more flexible, non-linear reasoning paths, significantly improving performance on challenging benchmarks.
The core innovation lies in leveraging the model’s inherent abstract reasoning to generate generalizable tools, combined with automatic error traceback-based correction. Experiments on the MATH and TabMWP datasets demonstrate that CREATOR achieves 59.7% and 94.7% accuracy respectively, surpassing existing chain-of-thought, program-of-thought, and tool-using baselines. Moreover, on the newly introduced Creation Challenge dataset, CREATOR attains an average of 75.5%, highlighting its capacity to handle novel, complex problems without relying on existing tools.
This approach not only boosts accuracy but also enhances knowledge transfer and adaptability across diverse tasks. The automatic rectification mechanism further improves robustness, increasing accuracy by approximately 10%. The framework’s ability to generate diverse, reusable tools marks a significant step toward autonomous AI systems capable of continuous self-improvement. While promising, challenges remain in reducing dependency on demonstrations and optimizing computational costs. Future work will focus on expanding tool diversity, improving correction mechanisms, and applying the framework across broader domains, ultimately pushing AI toward more self-reliant, flexible problem-solving.
Deep Dive
Abstract
Large Language Models (LLMs) have made significant progress in utilizing tools, but their ability is limited by API availability and the instability of implicit reasoning, particularly when both planning and execution are involved. To overcome these limitations, we propose CREATOR, a novel framework that enables LLMs to create their own tools using documentation and code realization. CREATOR disentangles abstract tool creation and concrete decision execution, resulting in improved performance. We evaluate CREATOR on MATH and TabMWP benchmarks, respectively consisting of challenging math competition problems and diverse tabular contents. Remarkably, CREATOR outperforms existing chain-of-thought, program-of-thought, and tool-using baselines. Additionally, we introduce the Creation Challenge dataset, featuring 2K diverse questions, to emphasize the necessity and benefits of LLMs' tool creation ability. Further research demonstrates that leveraging LLMs as tool creators facilitates knowledge transfer, and LLMs exhibit varying levels of tool creation abilities, enabling them to adapt to diverse situations. The tool creation ability revolutionizes the LLM's problem-solving paradigm, driving us closer to the next frontier of artificial intelligence. All the codes and data are released.