AIOS Compiler: LLM as Interpreter for Natural Language Programming and Flow Programming of AI Agents
Proposed CoRE system uses large language models as interpreters, unifying natural language, pseudo-code, and flow programming for AI agents.
Key Findings
Methodology
The approach introduces the CoRE language, structuring natural language instructions with step names, types (process, decision, terminal), and connections. It employs LLMs (e.g., GPT-4, Mixtral) as interpreters, executing instructions step-by-step. The system integrates external memory for context management and external tools for domain-specific tasks. The execution pipeline involves retrieving relevant info, constructing prompts with task context, invoking LLM responses, and determining subsequent steps based on step types and outputs. Validation on OpenAGI benchmarks shows superior performance over baseline methods, especially in complex multi-step tasks, demonstrating robustness and scalability across different LLMs.
Key Results
- On OpenAGI, CoRE with GPT-4 achieved over 92% task completion rate, outperforming Zero-shot and CoT strategies by 20-25%, with average scores surpassing 0.55 in metrics like BERT and CLIP scores.
- Using Mixtral, CoRE demonstrated consistent improvements, notably in task 2 (BERT score 0.2593), indicating strong cross-model robustness.
- Incorporating external tools for information retrieval and real-time data significantly enhanced accuracy, especially in tasks requiring domain knowledge or live data, validating the system’s practical effectiveness.
Significance
This work advances natural language programming by creating a unified, interpretable framework that lowers barriers for non-experts. By leveraging LLMs as interpreters and integrating external tools, it addresses key limitations in domain-specific knowledge and real-time information access. The approach fosters democratization of programming, enabling broader participation and automation in AI-driven applications. It paves the way for more flexible, explainable, and scalable intelligent agents capable of complex multi-step reasoning and decision-making, impacting industries from automation to customer service.
Technical Contribution
The core innovation lies in the design of the CoRE language, which unifies natural language, pseudo-code, and flow control within a structured syntax. The system employs prompt engineering to guide LLMs through stepwise execution, with mechanisms for external memory and tool invocation. This architecture enhances interpretability, flexibility, and task adaptability. The multi-model validation demonstrates the framework’s robustness, and the integration of external modules extends the model’s capabilities beyond static language understanding, enabling real-time data access and domain-specific operations. These contributions collectively push the frontier of natural language AI systems.
Novelty
This is the first work to unify natural language, pseudo-code, and flow programming under a single structured representation, leveraging LLMs as interpreters for multi-type program execution. Unlike prior approaches that rely solely on rule-based parsing or direct code generation, CoRE emphasizes logical structuring and stepwise interpretability, facilitating complex task handling and external tool integration. The innovative use of prompt engineering for dynamic execution and the multi-model validation further distinguish this work, offering a new paradigm for human-AI collaboration in programming.
Limitations
- The system depends on manually crafted instructions, which may limit scalability and efficiency. Ambiguities in natural language can still cause misinterpretation, especially in less structured inputs.
- High computational costs associated with large models like GPT-4 limit real-time deployment and accessibility. Response latency increases with task complexity.
- External tool invocation relies on API availability and interface design, which may restrict performance in certain domains or with unstandardized tools.
Future Work
Future research will focus on automating instruction generation, possibly through reinforcement learning or self-supervised methods, to reduce manual effort. Enhancing multi-modal input support, such as incorporating images or speech, will broaden applicability. Improving model efficiency and reducing inference costs are also priorities. Additionally, expanding the framework to handle more complex, real-world scenarios with adaptive learning capabilities will be explored, aiming for broader industrial deployment and user-friendly interfaces.
AI Executive Summary
Natural language programming has long promised to democratize coding, but its inherent ambiguity and verbosity have limited practical adoption. Traditional programming languages, while efficient, remain inaccessible to many due to syntax complexity. Recent advances in large language models (LLMs) like GPT-4 and Mixtral have opened new avenues, enabling machines to better understand and generate natural language. Building on this, the CoRE system introduces a unified framework that structures natural language instructions into a logical, programmable format.
At its core, CoRE develops a specialized language that unifies natural language, pseudo-code, and flow control constructs, allowing users to specify complex tasks in an intuitive manner. The system employs LLMs as interpreters, executing instructions step-by-step, guided by carefully designed prompts. It incorporates external memory to store intermediate results, reducing token load, and external tools to access real-time data or domain-specific knowledge. This hybrid approach ensures high accuracy and flexibility, overcoming the limitations of pure language models.
Experimental validation on the OpenAGI benchmark demonstrates that CoRE outperforms existing strategies like Zero-shot and Chain-of-Thought, achieving over 92% task completion rate with GPT-4, and maintaining robustness across different models. The results highlight its potential to facilitate complex multi-step reasoning, decision-making, and automation tasks, making AI more accessible and reliable.
This work significantly impacts both academia and industry by lowering the barrier to AI programming, enabling non-experts to create intelligent agents through natural language. It also opens pathways for integrating external knowledge sources, real-time data, and multi-modal inputs, paving the way for next-generation AI systems. Despite current limitations—such as reliance on manual instruction crafting and computational costs—the framework sets a foundation for future automation, scalability, and broader deployment in real-world applications, from smart assistants to automated workflows.
Deep Dive
Abstract
Since their inception, programming languages have trended towards greater readability and lower barriers for programmers. Following this trend, natural language can be a promising type of programming language that provides great flexibility and usability and helps towards the democracy of programming. However, the inherent vagueness, ambiguity, and verbosity of natural language pose significant challenges in developing an interpreter that can accurately understand the programming logic and execute instructions written in natural language. Fortunately, recent advancements in Large Language Models (LLMs) have demonstrated remarkable proficiency in interpreting complex natural language. Inspired by this, we develop a novel system for Code Representation and Execution (CoRE), which employs LLM as interpreter to interpret and execute natural language instructions. The proposed system unifies natural language programming, pseudo-code programming, and flow programming under the same representation for constructing language agents, while LLM serves as the interpreter to interpret and execute the agent programs. In this paper, we begin with defining the programming syntax that structures natural language instructions logically. During the execution, we incorporate external memory to minimize redundancy. Furthermore, we equip the designed interpreter with the capability to invoke external tools, compensating for the limitations of LLM in specialized domains or when accessing real-time information. This work is open-source at https://github.com/agiresearch/CoRE, https://github.com/agiresearch/OpenAGI, and https://github.com/agiresearch/AIOS.