Prompting Is Programming: A Query Language for Large Language Models

The evolution of NLP has seen large-scale pretraining of models like GPT, BERT, and T5, achieving state-of-the-art results in tasks such as translation, summarization, and question answering. Early prompt engineering involved crafting static prompts, but lacked flexibility and scalability. Recent advances introduced few-shot and meta-prompting techniques, improving adaptability. Tool use frameworks like LangChain and ReAct further enabled multi-step reasoning but faced efficiency bottlenecks. These methods still rely heavily on repeated model calls, which are costly and slow, especially as models grow larger. Consequently, optimizing inference efficiency and controllability remains a major challenge. This paper situates itself within this context, proposing a systematic programming paradigm to address these issues.

Despite progress, current prompting techniques are limited by static instructions, leading to high costs and limited control over outputs. Manual interaction, especially in complex multi-turn dialogues or task-specific workflows, is inefficient and error-prone. The inability to dynamically constrain outputs or optimize inference paths results in excessive model calls and unpredictable results. These issues hinder deployment at scale, particularly when using paid APIs with strict cost constraints. Therefore, a flexible, programmable approach that can automate and optimize inference while maintaining expressiveness is critically needed.

The core innovation is the development of LMQL, a high-level query language that transforms prompts into scripts with declarative constraints. It introduces formal semantics based on 'final' and 'follow' abstractions for partial evaluation, enabling automatic generation of token masks that prune the search space. LMQL’s syntax combines SQL-like declarative statements with Python-style scripting, allowing users to specify constraints such as maximum output length, entity restrictions, or logical conditions directly within prompts. This approach enables efficient, constrained decoding, reducing the number of expensive model calls. Unlike prior static prompt methods, LMQL provides a systematic, scalable framework for prompt automation and optimization across diverse models and tasks.

• �� Design LMQL syntax, integrating declarative constraints with scripting control flow.
• �� Implement 'final' and 'follow' abstractions to formalize partial evaluation semantics.
• �� Use these semantics to automatically generate model-specific token masks during decoding.
• �� Develop a runtime that applies constraints eagerly, pruning the search space dynamically.
• �� Support multiple decoding strategies (argmax, beam, sampling) within the framework.
• �� Ensure model-agnostic compatibility, enabling deployment across different Transformer-based models.
• �� Validate the approach through extensive experiments on tasks like question answering, code generation, and dialogue, comparing costs, speed, and accuracy against baselines.

The evaluation involved datasets such as SQuAD for question answering, CodeX for code synthesis, and synthetic multi-turn dialogues. Baselines included standard prompt methods, LangChain, and ReAct. Metrics assessed were accuracy, inference cost, and latency. Experiments tested various constraints, such as output length limits and entity restrictions, to demonstrate pruning effectiveness. Ablation studies analyzed the impact of different components like mask generation and control flow. Results confirmed that LMQL reduces inference costs by 26%-85%, with over 2x speedup, while maintaining or improving performance. Cross-task robustness and scalability were also verified.

LMQL consistently outperformed baseline prompting methods, achieving up to 85% reduction in inference costs and doubling inference speed. It maintained high accuracy across tasks, with minimal performance degradation. The automatic mask generation effectively eliminated invalid or redundant decoding paths, leading to more reliable outputs. Ablation studies highlighted the importance of constraints and partial evaluation in optimizing efficiency. The framework demonstrated adaptability to various models and tasks, confirming its generality and practical value.

LMQL enables cost-efficient deployment of large models in enterprise NLP pipelines, content moderation, and automated coding. Its scripting and constraint capabilities make it suitable for interactive AI assistants, chatbots, and multi-turn dialogue systems requiring high controllability. The approach can be integrated into existing ML workflows, reducing operational costs and latency. Long-term, LMQL supports autonomous AI systems capable of self-optimizing inference strategies, facilitating scalable AI solutions for industry and research.

Current implementation depends on predefined constraints and scripts, which may require manual tuning for complex scenarios. Mask generation for ultra-large models (e.g., GPT-4) incurs computational overhead, limiting real-time applications. The framework’s reliance on model internals like tokenization may restrict cross-model generality. Further research is needed to automate constraint inference, improve scalability, and extend multi-modal support. Addressing these limitations will be crucial for broader adoption and robustness.