An LLM Compiler for Parallel Function Calling

The evolution of LLMs from simple content generators to reasoning engines has been marked by significant milestones, including GPT series, LLaMA, and PaLM. Early work focused on single-task performance, but as reasoning capabilities improved, the need for multi-tool integration emerged. Frameworks like ReAct introduced tool invocation, enabling models to call external functions for reasoning enhancement. However, these approaches predominantly used serial execution, which limited scalability due to increased latency and costs. Recent efforts explored parallelism but lacked systematic optimization, especially for complex dependency management. The growing demand for open-source models and parameter-efficient training techniques further highlighted the need for scalable, flexible orchestration frameworks that can handle diverse tasks efficiently.

The core challenge lies in the serial nature of existing function calling methods, which severely hampers inference speed and increases operational costs. Complex tasks often involve multiple interdependent steps, making manual dependency management impractical. Current solutions lack automated dependency analysis and dynamic re-planning, leading to redundant computations and suboptimal resource utilization. This bottleneck restricts the deployment of large models in real-time applications like automated reasoning, multi-modal integration, and decision support systems. Overcoming these limitations requires a framework capable of automatically analyzing task dependencies, enabling high concurrency, and adapting to dynamic environments.

The paper introduces a compiler-inspired framework, LLMCompiler, which innovatively applies classical compiler optimization strategies to LLM function calling. Its key innovations include:

1) Automatic dependency graph generation using LLM reasoning, enabling precise identification of task dependencies.

2) Parallel scheduling based on dependency DAGs, maximizing concurrency and reducing latency.

3) Support for dynamic re-planning, allowing the system to adapt to intermediate results and changing dependencies.

4) Compatibility with multiple models and tools, including open-source options like LLaMA-2, broadening applicability.

These innovations collectively address the inefficiencies of serial execution, enabling scalable, cost-effective, and accurate multi-tool reasoning.

• �� Input natural language task is processed by the Function Calling Planner, which uses LLM prompting to generate a dependency DAG, identifying tasks and their dependencies.
• �� The Task Fetching Unit monitors the DAG, dispatching independent tasks immediately and queuing dependent tasks until dependencies are resolved.
• �� The Executor asynchronously executes tasks using specified tools or models, storing intermediate results.
• �� During execution, the system monitors intermediate outputs, and if necessary, triggers re-planning by regenerating the dependency DAG based on new information.
• �� The entire process supports multi-model and multi-tool integration, with optional in-context examples to improve planner accuracy.
• �� The framework is optimized for streaming dependency graphs, reducing latency by allowing tasks to start as soon as dependencies are met.

The evaluation involved datasets like HotpotQA, Movie Recommendation, ParallelQA, and WebShop, using GPT-3.5-turbo and LLaMA-2 70B. Baselines included ReAct and OpenAI's parallel function calling. Metrics measured were latency, cost, and accuracy. Experiments tested simple parallel patterns, complex dependency graphs, and dynamic re-planning scenarios. Ablation studies analyzed the impact of dependency analysis, streaming, and re-planning. Results consistently showed significant improvements in speed and cost, with accuracy gains up to 9%. The framework demonstrated robustness across models and task complexities, validating its generality and scalability.

LLMCompiler achieved up to 3.7× latency speedup and 6.7× cost reduction over ReAct in benchmark tasks. In complex scenarios like ParallelQA, speedup reached 2.27× with 4.65× cost savings. Accuracy improved by approximately 9%, especially in tasks with dynamic dependencies. Combining with open-source LLaMA-2, it maintained performance gains, confirming broad applicability. The experiments also highlighted the effectiveness of streaming dependency graphs, reducing latency by up to 1.3× in certain benchmarks. These results demonstrate the framework’s capacity to handle diverse, real-world reasoning tasks efficiently.

The framework is suitable for multi-step reasoning, automated question answering, knowledge graph construction, and decision support systems. It enables enterprises to deploy large models more efficiently by reducing inference costs and latency. Future applications include real-time AI assistants, autonomous agents, and complex multi-modal reasoning systems. Its ability to dynamically re-plan makes it adaptable to changing environments, supporting applications in robotics, finance, and healthcare where timely decision-making is critical.

Current reliance on predefined tools and prompt examples may limit flexibility in unforeseen tasks. Handling extremely complex or highly dynamic dependency graphs can introduce scheduling overhead, impacting performance. High computational resource requirements pose deployment challenges, especially in large-scale settings. Future work should focus on adaptive scheduling algorithms, reinforcement learning-based optimization, and broader multi-model integration to address these issues and enhance robustness.