CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
CODEGEN uses multi-turn prompts to improve synthesis, reaching 47.34% on MTPB with its 16.1B model.
Key Findings
Methodology
The paper trains autoregressive Transformer CODEGEN models with 350M, 2.7B, 6.1B, and 16.1B parameters using next-token prediction. Models are trained sequentially on THEPILE, BIGQUERY, and BIGPYTHON, producing CODEGEN-NL, -MULTI, and -MONO variants. The central synthesis paradigm factorizes a complex specification into natural-language turns, with each turn generating a subprogram conditioned on prior prompts and outputs. JAXFORMER enables TPU-v4 training.
Key Results
- On the 164-task HumanEval benchmark, CODEGEN-MONO 16.1B obtains pass@1=29.28%, pass@10=49.86%, and pass@100=75.00%; the 6.1B model obtains 26.13%, 42.29%, and 65.82%, showing gains from both scale and Python-specific data.
- On the 115-problem MTPB benchmark, CODEGEN-MONO 16.1B reaches 47.34%, versus 26.27% for CODEGEN-MULTI 16.1B and 30.33% for CODEGEN-NL 16.1B. Python specialization substantially improves multi-turn synthesis.
- Replacing concatenated single-turn specifications with multi-turn prompts reduces perplexity from 10.25 to 8.05 and raises pass rate from 38.74% to 47.34% for the 16.1B model; the 350M model improves from 5.75% to 16.98%.
Significance
CODEGEN lowers the access barrier to competitive code models by releasing checkpoints, training infrastructure, and a benchmark rather than requiring researchers to depend on closed Codex systems. Conceptually, it reframes synthesis as collaborative, incremental construction instead of one-shot completion. The results suggest that user-interface structure is itself a performance variable: decomposing intent can reduce search complexity and make specifications more compatible with learned comment-code patterns. This has implications for developer tools, education, and human-AI programming workflows.
Technical Contribution
The primary contribution is empirical and infrastructural rather than a new decoder. CODEGEN studies how model size, language-data composition, and sequential training affect synthesis. Its THEPILE→BIGQUERY→BIGPYTHON curriculum provides natural-language pretraining, multilingual code exposure, and Python specialization. JAXFORMER supports TPU-v4 data and model parallelism. MTPB formalizes at least three turns with cross-turn variable or function dependencies, then evaluates concatenated programs in an isolated Python environment using output capture and type-relaxed comparison.
Novelty
The work is the first to systematically quantify multi-turn program synthesis in large autoregressive language models and to release an open benchmark for it. Unlike HumanEval, which supplies a function signature and evaluates single-turn completion, MTPB requires programs to be generated from scratch, state to be maintained across turns, and the final state to be printed. Its fundamental novelty is treating intent factorization as a controllable source of improvement, alongside scaling model and data size.
Limitations
- MTPB contains only 115 expert-authored problems and five test-case sets per problem. Although categories are balanced, this scale cannot represent the boundary behavior and maintenance complexity of real software.
- The conclusions are Python-, template-, sampling-, and environment-dependent. Errors can propagate across turns, and the study does not establish equivalent gains on large repositories or production-level engineering tasks.
- GitHub-derived data may contain duplication, inaccurate comments, license ambiguity, and contamination. The weak alignment between comments and code also makes the source of emergent multi-turn ability difficult to isolate.
Future Work
Future work should expand MTPB across languages, domains, and stronger test suites; study automatic task decomposition, execution feedback, and conversational repair; and compare retrieval augmentation with search-based decoding and supervised alignment. Better deduplication, license auditing, sandboxing, and repository-level metrics are needed to move CODEGEN from benchmark completion toward dependable collaborative software development.
AI Executive Summary
Program synthesis is constrained by two opposing difficulties: the space of possible programs is enormous, while users often struggle to express a complete specification. Codex demonstrated that large language models can generate Python from natural language, but expensive training and closed access limited reproducibility. Salesforce Research addresses this gap with CODEGEN, an open family ranging from 350M to 16.1B parameters, together with the JAXFORMER training library.
CODEGEN is an autoregressive Transformer trained sequentially on THEPILE, BIGQUERY, and BIGPYTHON. The paper’s central interaction idea is multi-turn synthesis: instead of asking for a complete program in one prompt, a user describes successive subproblems and the model generates code that reuses earlier variables and functions. The authors introduce MTPB, a benchmark of 115 multi-step tasks, and execute concatenated outputs in an isolated Python environment.
The 16.1B CODEGEN-MONO model scores 29.28% pass@1, 49.86% pass@10, and 75.00% pass@100 on HumanEval, while reaching 47.34% on MTPB. For the same intent, multi-turn factorization raises the 16.1B pass rate from 38.74% to 47.34% and lowers prompt perplexity from 10.25 to 8.05. The study therefore links synthesis quality not only to parameters and code data, but also to how specifications are structured. Its evidence remains limited by the small, Python-only MTPB and the gap between benchmark tasks and real software engineering.
Deep Analysis
Background
Program synthesis aims to generate executable programs from natural language, examples, or formal specifications. Domain-specific languages reduce search but limit generality; Transformer language models such as GPT-NEO, GPT-J, and Codex instead learn broad distributions from large corpora. HumanEval established a practical zero-shot Python benchmark, yet open models competitive with Codex and quantitative tests of multi-turn synthesis remained scarce.
Core Problem
Given a natural-language intent, a model must find a functionally correct program in a vast search space. Formal specifications are costly for users, while isolated examples may under-specify behavior. Long prompts also force the model to track dependencies among many operations. The paper asks whether scale and code-data composition produce multi-turn ability, and whether factorizing one intent improves synthesis over a concatenated single turn.
Innovation
- �� Releases CODEGEN-NL, -MULTI, and -MONO plus JAXFORMER.
- �� Introduces MTPB with 115 expert-written tasks decomposed into at least three turns.
- �� Uses prompt perplexity as a proxy for intent understanding and compares factorized versus concatenated prompts.
- �� Evaluates executable functional correctness rather than surface similarity, including cross-turn state reuse and final output capture.
Methodology
- �� Data: THEPILE is an 825.18 GiB English corpus; BIGQUERY supplies C, C++, Go, Java, JavaScript, and Python; BIGPYTHON contains permissively licensed GitHub Python code.
- �� Training: autoregressive Transformers predict the next token at 350M–16.1B parameters; initialization proceeds THEPILE→BIGQUERY→BIGPYTHON.
- �� Sampling: HumanEval uses nucleus sampling with top-p=0.95 and temperatures 0.2, 0.6, and 0.8; MTPB conditions each turn on interleaved history.
- �� Evaluation: generated code is concatenated and executed in isolation; the final print output is captured and compared with gold answers using type-relaxed equivalence.
Experiments
HumanEval contains 164 handwritten Python problems and compares GPT-NEO, GPT-J, Codex, and CODEGEN using pass@1/10/100. MTPB contains 115 tasks, five test cases per task, and 40 samples per case. Experiments vary model size, code-data size, and language specialization, and compare multi-turn prompts with concatenated single-turn counterparts using perplexity and pass rate. Difficulty-level analysis is also reported.
Results
On HumanEval, CODEGEN-NL 16.1B scores 14.24/23.46/38.33, CODEGEN-MULTI 16.1B scores 18.32/32.07/50.80, and CODEGEN-MONO 16.1B scores 29.28/49.86/75.00 for pass@1/10/100. MTPB scores for MONO models are 16.98%, 38.72%, 43.52%, and 47.34%. Multi-turn formulation improves the four model sizes by 11.23, 13.29, 15.04, and 8.60 percentage points over single-turn formulation.
Applications
The approach can support natural-language scripting, data cleaning, educational programming, rapid prototyping, and interactive coding assistants. Multi-turn interaction is particularly suitable for decomposing imports, transformations, checks, and outputs. Deployment requires sandbox execution, static analysis, generated tests, license auditing, and human review because functional benchmarks do not guarantee security, efficiency, or maintainability.
Limitations & Outlook
Training is computationally expensive and JAXFORMER targets TPU-v4. GitHub data may contain duplicates, incorrect comments, and licensing issues. MTPB covers only Python and 115 tasks, with limited test inputs; it does not measure cross-file interfaces, concurrency, performance, or maintenance in large repositories. Future work should add multilingual and repository-level evaluation, compiler feedback, automatic decomposition, retrieval, and execution-guided repair.
Plain Language Accessible to non-experts
Imagine CODEGEN as a kitchen assistant trained on an enormous collection of recipes. THEPILE teaches ordinary human language; BIGQUERY exposes it to recipes written in several programming languages; BIGPYTHON gives it concentrated practice in Python. The assistant is not memorizing one dish but learning patterns for turning instructions into procedures.
A single request such as “prepare a complete dinner, clean the ingredients, and serve the result” may be overwhelming. Multi-turn synthesis breaks it into smaller instructions: gather tools, prepare ingredients, cook them, and plate the dish. After each step, the assistant keeps the result on the counter so the next step can use it. MTPB is a 115-task kitchen examination built around this idea.
The largest Python model succeeds on 29.28% of HumanEval tasks with one attempt and reaches 75.00% when allowed 100 samples. On the multi-turn test it reaches 47.34%. Turning one large request into several smaller ones improves its score by 8.60 percentage points. Yet benchmark dishes are simpler than real kitchens: generated programs still need testing, isolation, safety checks, and human supervision.
ELI14 Explained like you're 14
Think of CODEGEN as a robot teammate in a game where the mission is to write programs. You could say, “Build a tool that finds an email, cleans the name, and prints it.” But if the robot tries everything at once, it may forget an earlier step. The multi-turn strategy turns the mission into levels: import a tool, find the email, remove extra text, and print the answer.
Before playing, the robot studies lots of ordinary writing and code in C, C++, Go, Java, JavaScript, and Python. Then it gets extra Python practice. Its parameters are like tiny pattern-recognition parts: more parts and better practice usually help, although bigger does not mean perfect.
The biggest Python model gets about 29% of HumanEval tasks right on its first try and 75% when it can offer 100 guesses. In the 115-task MTPB multi-turn test, it succeeds 47.34% of the time. Splitting the mission raises the score from 38.74% to 47.34%. Pretty useful, right? But it can still write unsafe or buggy code, so real developers must test and check it.
The coolest lesson is that how you ask can matter almost as much as how smart the robot is. A huge wall of instructions is hard to follow; a sequence of clear mini-missions is easier. This does not mean the robot understands like a human. It predicts likely next pieces from training patterns. Its outputs can look confident while being wrong, so code assistants should be treated like fast teammates—not unquestionable experts!
Glossary
Autoregressive Transformer
A neural architecture that predicts the next token from previous tokens. Transformer attention lets it integrate information across long contexts.
The core architecture and learning objective of CODEGEN.
Pass@k
The estimated probability that at least one of k generated samples passes all tests. It measures functional correctness rather than textual similarity.
The principal metric on HumanEval.
Prompt Perplexity
A measure of the model’s uncertainty about a prompt; lower values generally indicate better compatibility with learned patterns. It is not itself a correctness score.
Used as a proxy for understanding user intent.
Multi-Turn Program Synthesis
Generating one program through multiple prompts and subprograms while retaining prior state. It allows complex intent to be expressed incrementally.
The paper’s central interaction paradigm.
MTPB
The Multi-Turn Programming Benchmark contains 115 expert-authored, multi-step Python problems. Success is determined by executing generated code and checking outputs.
The first open quantitative benchmark for this capability.
JAXFORMER
A JAX-based training library designed for large models on TPU-v4. It supports data and model parallelism.
The open infrastructure used to train and release CODEGEN.
Open Questions Unanswered questions from this research
- 1 It remains unclear whether multi-turn gains transfer to large repositories, cross-file dependencies, and long-term maintenance. Repository-level datasets, continuous tests, and developer-centered evaluations are needed.
- 2 Prompt perplexity correlates with success, but its predictive reliability is unknown. Future studies should combine it with compilation traces, execution feedback, and error-type analysis.
- 3 The relative roles of scale, data structure, and sequential training remain unresolved because GitHub comments provide weak and noisy supervision for multi-turn behavior.
Applications
Immediate Applications
Interactive Python assistant
Developers can split a request into imports, transformations, validation, and output, allowing CODEGEN to produce and reuse code incrementally. Sandbox execution, unit tests, static scanning, and review are required before deployment.
Programming education
Instructors can ask students to describe an algorithm step by step, then compare generated subprograms with test results. MTPB-style exercises teach decomposition and debugging, but teachers must verify correctness and safety.
Long-term Vision
Human-AI software engineering
A future assistant could remember interfaces, tests, and failures while implementing cross-file features collaboratively. Realizing this vision requires stronger retrieval, execution feedback, license governance, and reliability evaluation.
Abstract
Program synthesis strives to generate a computer program as a solution to a given problem specification, expressed with input-output examples or natural language descriptions. The prevalence of large language models advances the state-of-the-art for program synthesis, though limited training resources and data impede open access to such models. To democratize this, we train and release a family of large language models up to 16.1B parameters, called CODEGEN, on natural language and programming language data, and open source the training library JAXFORMER. We show the utility of the trained model by demonstrating that it is competitive with the previous state-of-the-art on zero-shot Python code generation on HumanEval. We further investigate the multi-step paradigm for program synthesis, where a single program is factorized into multiple prompts specifying subproblems. To this end, we construct an open benchmark, Multi-Turn Programming Benchmark (MTPB), consisting of 115 diverse problem sets that are factorized into multi-turn prompts. Our analysis on MTPB shows that the same intent provided to CODEGEN in multi-turn fashion significantly improves program synthesis over that provided as a single turn. We make the training library JAXFORMER and model checkpoints available as open source contribution: https://github.com/salesforce/CodeGen.