Compile by Training: Turning Natural-Language Specifications into Local Neural Functions
Proposes 'Compile by Training' to convert natural language specs into local neural functions with 83.6% accuracy, using teacher models and adapter fine-tuning.
Key Findings
Methodology
The approach leverages large teacher models to generate task-specific examples, which are used to fine-tune lightweight LoRA adapters within a shared interpreter framework. The process involves initial fast prediction via Program-as-Weights (PAW), followed by iterative gradient-based optimization to specialize adapters. The final program packages include adapter parameters, prompts, and metadata, enabling local, versioned, and composable execution without ongoing teacher calls. This system achieves 83.6% semantic accuracy on FuzzyBench-Hard, outperforming fast compilation baselines.
Key Results
- On FuzzyBench-Hard, the trained model reaches 83.6% semantic accuracy, a 59-point increase over the 0.224 baseline of the fast compiler, demonstrating significant improvements in complex fuzzy tasks. The training process takes about one minute, making it suitable for interactive deployment. Multiple real-world applications, including multi-site web assistants, language-controlled 3D avatars, and English-Claudish bidirectional translation, showcase the system's versatility and robustness across diverse scenarios.
Significance
This work addresses the challenge of frequent, cost-intensive remote model calls by enabling local, reusable neural functions derived from natural language specifications. It advances the field by combining large-scale example synthesis with parameter-efficient adaptation, paving the way for scalable, customizable AI components that can be integrated into various applications, reducing latency and dependency on external providers.
Technical Contribution
The core innovation lies in integrating large model-generated examples with gradient-based adapter training within the PAW framework, enabling rapid, high-accuracy compilation of neural functions. The approach introduces a systematic pipeline from specification to packaged executable, supporting versioning, caching, and composition. It extends prior work on amortized prediction by investing additional compute at compile time for better task-specific performance, thus balancing speed and accuracy effectively.
Novelty
This is the first work to combine large language model-generated supervision with adapter fine-tuning for task-specific neural function compilation, achieving high accuracy in complex fuzzy tasks. Unlike prior rule-based or single-pass prediction methods, this approach emphasizes training-based specialization, enabling high fidelity and local execution, which is a significant step forward in neural program synthesis.
Limitations
- The training process depends on the quality and diversity of teacher-generated examples, which may inherit biases or errors, affecting robustness. The compile time, though reduced to about a minute, still exceeds real-time constraints for some applications. The system's performance on extremely complex or novel tasks remains limited, and hardware requirements for large-scale synthesis are non-trivial.
Future Work
Future directions include improving the efficiency of example synthesis, exploring reinforcement learning or self-supervised signals to enhance robustness, and extending multi-modal capabilities to incorporate images, audio, and video. Further research will aim to reduce training time, improve generalization to unseen tasks, and develop more scalable, adaptive systems for real-world deployment.
AI Executive Summary
The rapid growth of large-scale language models has revolutionized NLP, but deploying these models repeatedly for individual tasks remains costly and slow. Traditional solutions involve either rule-based systems or calling remote APIs, each with limitations in flexibility and efficiency. To address this, the authors introduce 'Compile by Training,' a novel paradigm that transforms natural language task specifications into local, reusable neural functions. This approach leverages large teacher models to generate synthetic examples, which are then used to fine-tune lightweight adapters within a shared interpreter framework. The process combines the speed of Program-as-Weights predictions with the accuracy gains from targeted training, resulting in a system capable of producing functions with 83.6% semantic correctness on challenging benchmarks.
The core technical innovation is the integration of large model-generated supervision with parameter-efficient adaptation, enabling rapid, high-fidelity compilation. The training pipeline involves initial fast prediction, iterative fine-tuning, and packaging of the resulting program, which can be executed locally without ongoing external calls. This setup supports version control, caching, and composition, making it suitable for real-world applications.
Experimental results demonstrate that, although the training process takes about a minute, the resulting functions outperform fast compilation baselines by a large margin, especially on complex fuzzy tasks. The system has been deployed in multiple scenarios, including web assistants, 3D avatar control, and language translation, validating its versatility and practical value.
This work significantly advances the field by providing a scalable, customizable solution for neural program synthesis, reducing reliance on costly remote models, and enabling local, efficient, and accurate task execution. Future research will focus on further reducing training time, enhancing robustness, and expanding multi-modal capabilities, aiming to make neural functions an integral part of everyday intelligent systems.
Deep Analysis
Background
The evolution of NLP has been driven by large pre-trained models like GPT, PaLM, and Qwen, which demonstrate remarkable capabilities in understanding and generating natural language. Early methods relied heavily on rule-based systems or remote API calls, which limited scalability and increased costs. Recent advances include parameter-efficient fine-tuning techniques such as LoRA, prefix tuning, and prompt tuning, enabling task adaptation with minimal parameters. Despite these developments, deploying large models repeatedly remains resource-intensive, especially for frequent or latency-sensitive tasks. The challenge is to develop methods that can generate local, reusable neural functions from natural language descriptions, balancing accuracy, efficiency, and flexibility. Prior work on neural program synthesis and instruction tuning laid the groundwork but lacked scalable, high-accuracy solutions for complex fuzzy tasks, motivating this research.
Core Problem
The core issue is how to efficiently convert natural language specifications into local neural functions suitable for deployment in real-world applications. Existing approaches either incur high costs by calling large models repeatedly or rely on rule-based systems that lack flexibility. Fine-tuning large models for each task is computationally expensive and impractical at scale. Moreover, achieving high semantic accuracy in fuzzy, ambiguous tasks remains difficult. The key bottleneck is designing a system that can synthesize task-specific, high-fidelity functions quickly enough for interactive use, while maintaining the benefits of neural generalization and composability.
Innovation
This paper introduces 'Compile by Training,' which combines large model synthesis with parameter-efficient adaptation. The main innovations include:
1) Using large teacher models to generate synthetic input-output examples from natural language specs, enriching training data.
2) Employing a shared frozen interpreter (Qwen3-0.6B) with lightweight LoRA adapters, enabling task-specific specialization.
3) Integrating an amortized PAW predictor for rapid initialization, followed by gradient-based fine-tuning to improve accuracy.
4) Packaging the trained adapter, prompt scaffold, and metadata into a reusable program artifact.
These innovations enable fast (around one minute) compilation with high semantic correctness, supporting versioning and composition, and significantly improving over prior single-pass prediction methods.
Methodology
- �� User submits a natural language specification for a task.
- �� Teacher models (gpt-4o, GPT-5.5) synthesize input-output pairs, validated and filtered.
- �� The initial adapter parameters and prompt scaffold are predicted via PAW, providing a starting point.
- �� Gradient descent fine-tunes the adapter on the synthesized dataset, minimizing the loss L(θs) = Σ(x,y)∈Ds -log pθs (y | rs(x)).
- �� The trained adapter, scaffold, and metadata are bundled into a program artifact.
- �� At runtime, the local interpreter combines new inputs with the packaged program, executing without external calls.
- �� The system supports version control, caching, and asynchronous compilation, enabling seamless deployment.
Experiments
The evaluation used FuzzyBench-Hard, a challenging subset with no exact matches from PAW fast compiler. Results showed a jump from 0.224 to 0.836 in mean LEM accuracy, with training taking about 50 seconds. Ablation studies varied teacher models and data sizes, confirming the importance of synthetic supervision. Deployment in real applications, including multi-site web helpers, 3D avatars, and translation tools, demonstrated high responsiveness and robustness. The system maintained low latency (around 1 second queue wait) under load, validating its practicality for interactive use. The experiments confirmed that training-based compilation significantly outperforms fast prediction in complex fuzzy tasks.
Results
The trained models achieved 83.6% semantic accuracy on FuzzyBench-Hard, a substantial improvement over the 0.224 baseline. Training duration was approximately one minute, suitable for interactive deployment. In multi-application scenarios, the system effectively supported web assistants, 3D character control, and bidirectional language translation, with high responsiveness and accuracy. The approach demonstrated robustness across different teacher models and data scales, with accuracy improving as data increased. The results validate the effectiveness of combining synthetic supervision with parameter-efficient fine-tuning for neural function compilation.
Applications
This technology enables rapid creation of task-specific neural functions for diverse applications such as intelligent web assistants, virtual character control, and language translation. It reduces dependency on remote models, lowers operational costs, and improves response times. The system's modularity and versioning facilitate integration into larger software ecosystems, supporting dynamic updates and customization. Future expansion includes multi-modal inputs, enabling applications in video, audio, and sensor data processing, further broadening its industry impact.
Limitations & Outlook
Despite its advantages, the method relies heavily on the quality of teacher-generated examples, which may inherit biases or errors. The training process, though fast, still takes about a minute, limiting real-time adaptability in some scenarios. Performance on highly novel or complex tasks not covered by training data remains uncertain. Hardware requirements for large-scale synthesis and fine-tuning are non-trivial, and further optimization is needed for ultra-low latency applications. Addressing these limitations will be key to broader adoption.
Plain Language Accessible to non-experts
想象你在厨房里做饭，平时每次都要按照菜谱一步步操作，既麻烦又费时间。现在，你有一个聪明的助手，只要你告诉他“做番茄炒蛋”，他就能根据之前学到的做菜经验，自己准备所有材料和步骤。这个助手是通过看很多你平时做菜的视频（示例）学会的，然后用很少的调整就能帮你做出菜来。以前，你得每次都教他每一步，现在只要一句话，他就能自己完成。这个方法就像让大模型帮你“学会”任务，然后把它变成一个可以在本地快速运行的小程序，不用每次都依赖远程大模型，既快又省钱。
ELI14 Explained like you're 14
想象你有个超级厉害的朋友，他可以帮你写作业、画画或者玩游戏，只要你告诉他你想做什么。他平时会看很多视频和书，学会了很多技能。每次你说“帮我画一只飞翔的龙”，他就能用自己学到的东西，快速帮你完成。以前，你得花很多时间教他每个步骤，但现在，他自己就知道怎么做了。这就像用大模型教会一个小助手，然后让它自己操作，既快又方便。这种方法让我们的电脑变得更聪明，也能帮我们做很多事情，就像一个贴心的帮手一样。
Abstract
Many recurring text functions are easy to describe but difficult to implement with rules, while calling a large remote model for every input introduces repeated cost, latency, and dependency on a provider. We present compile by training, which turns a natural-language specification into a reusable neural function. At compile time, teacher models generate task-specific examples that are used to train a small adapter for a compact interpreter. The resulting function runs without the teachers and can be stored, versioned, and composed like ordinary software. On FuzzyBench-Hard, a subset on which the Program-as-Weights fast compiler produced no exact matches, compile by training reaches 83.6% semantic accuracy. This higher accuracy comes with a higher compile-time cost: roughly a minute rather than seconds for the fast compiler. We deploy the compiler in a public interactive service and demonstrate compiled functions in a multi-site website helper, a language-controlled 3D avatar, and a bidirectional English-Claudish translator.