PanGu-Coder: Program Synthesis with Function-Level Language Modeling
PanGu-Coder uses two-stage function-level training, reaching 17.07% HumanEval pass@1 with 317M parameters.
Key Findings
Methodology
PanGu-Coder adopts PanGu-α’s unidirectional decoder-only Transformer with an additional Query Layer, where the next-position embedding queries top-layer representations. Stage 1 applies Causal Language Modeling (CLM) to raw Python data for 188B tokens. Stage 2 uses loosely aligned docstring–function pairs and combines CODE-CLM with Docstr-MLM or Docstr-MCLM, explicitly emphasizing conditional generation from natural-language descriptions to Python function bodies.
Key Results
- On HumanEval, the 317M model obtains pass@1/10/100 of 17.07/24.05/34.55%, exceeding CodeX 300M at 13.17/20.37/36.27% on pass@1 and pass@10. The 2.6B model reaches 23.78/35.36/51.24%, with pass@1 above CodeX 2.5B’s 21.36%.
- On MBPP, the 317M and 2.6B models achieve pass@1 of 16.20% and 23.00%, respectively. The 2.6B system surpasses 6.7B InCoder’s 19.40% despite using a 1,024-token context, 147GB of data, and substantially fewer parameters.
- Evaluation samples 200 programs per problem and uses Chen et al.’s unbiased pass@k estimator. Performance is strongest at low k; CodeGen Mono performs better at pass@100, plausibly because its longer context and larger training corpus support longer solutions.
Significance
The work shifts code-generation evaluation toward executable functional correctness rather than surface-level similarity. It shows that medium-sized models can compete with much larger systems when training data are reorganized around functions and objectives are aligned with the generation interface. This is important for resource-efficient deployment in IDEs, educational tools, and enterprise coding systems. More broadly, it challenges the assumption that scaling parameters, data, and context is always the dominant route to better program synthesis.
Technical Contribution
The paper contributes four linked design choices: applying the PanGu-α Query Layer to program synthesis; separating broad CLM pretraining from function-level alignment training; defining CODE-CLM, Docstr-MLM, and Docstr-MCLM objectives over distinct subsequences; and constructing data through Python AST extraction, filtering, and function-body deduplication. It also evaluates executable behavior with temperature and nucleus sampling and mitigates missing-dependency artifacts through automatic imports, making functional evaluation more faithful to the intended function-level setting.
Novelty
The novelty is not the first use of a Transformer for code generation. Rather, PanGu-Coder turns noisy online repositories into function-level natural-language–code pairs and redesigns pretraining around the exact input/output structure of text-to-code synthesis. Compared with GPT-style continuous CLM, it explicitly prioritizes code continuation while treating the description with masked objectives. This provides a practical bridge between scalable decoder-only pretraining and structured conditional generation.
Limitations
- The released system supports only English prompts and Python outputs. Function-isolated training frequently omits imports, project state, configuration, and cross-file APIs; automatic library imports improve evaluation but do not reproduce real software environments.
- The 1,024-token context window can constrain long descriptions and implementations. This likely contributes to lower pass@100 than CodeGen Mono, whose training used longer context and much larger data and token budgets.
Future Work
Future work should extend the approach to multiple programming languages, repository-level context, and realistic dependency environments. The paper’s PanGu-Coder-FT direction—fine-tuning on competitive-programming problems and code with continuous-integration tests—could be combined with retrieval, execution feedback, longer contexts, and explicit loss weighting. Such systems should also evaluate security, maintainability, and robustness, not only test-case success.
AI Executive Summary
Program synthesis asks a model to turn a natural-language requirement into executable code. Existing systems such as CodeX and CodeGen benefit from enormous corpora and long contexts, yet textual similarity does not guarantee correctness. PanGu-Coder instead targets function-level synthesis: given an English docstring and Python signature, it generates a function body, which is judged by hidden tests in HumanEval and MBPP.
Built on the PanGu-α decoder-only Transformer, the model adds a Query Layer that uses the next-position embedding to attend over top-layer states. Training has two stages. Stage 1 applies CLM to raw GitHub Python data for 188B tokens, learning general program structure. Stage 2 uses AST-extracted docstring–function pairs and combines CODE-CLM with Docstr-MLM or Docstr-MCLM, aligning optimization with description-to-code generation. Filtering and deduplication leave approximately 147GB of pretraining data.
The 317M model reaches 17.07% HumanEval pass@1, compared with 13.17% for CodeX 300M; the 2.6B model reaches 23.78%, above CodeX 2.5B’s 21.36%. On MBPP, the 2.6B model obtains 23.00% pass@1, exceeding 6.7B InCoder’s 19.40%. Its weaknesses are equally instructive: a 1,024-token context, missing project dependencies, and weaker high-k performance. The study therefore argues for better data structure and objective design, not scaling alone.
Deep Analysis
Background
GPT, GPT-Neo, CodeX, CodeGen, AlphaCode, and InCoder established large-scale neural code generation. However, code corpora are noisy, contexts are expensive, and functional correctness remains difficult. HumanEval and MBPP evaluate hidden-test success. PanGu-Coder adapts PanGu-α to English-to-Python function synthesis and studies whether function-level alignment can deliver competitive results with less data and context.
Core Problem
Given a natural-language description, function signature, and optional examples, the model must generate a function body that passes unseen unit tests. The bottlenecks are semantic interpretation, algorithm selection, syntactic validity, dependency handling, long-range program structure, and weak supervision from inconsistent online docstrings. These challenges make token-level likelihood an imperfect proxy for executable correctness.
Innovation
- ��AST-based extraction creates structured <descr>, <python>, and <eoc> instances.
- ��Stage 1 uses continuous CLM for broad code knowledge; Stage 2 feeds each aligned pair separately.
- ��CODE-CLM focuses loss on generated code, while Docstr-MLM and Docstr-MCLM model the description with masking.
- ��The PanGu-α Query Layer and nucleus sampling provide the architectural and decoding mechanisms for efficient generation and functional evaluation.
Methodology
- ��Data: GHTorrent-associated GitHub repositories yielded about 65M Python files; deduplication and quality filters reduced the corpus to roughly 147GB.
- ��Representation: SentencePiece vocabulary was reduced to about 42K subtokens. Models contain 317M or 2.6B parameters and use a 1,024-token context.
- ��Stage 1: random initialization, Adam with β1=0.9, β2=0.95, weight decay 0.01, and CLM for 188B tokens.
- ��Stage 2: remove docstrings shorter than 19 words, bodies longer than 400 words, or length ratios above 32; continue for up to 1M steps, about 42B tokens.
- ��Generation: temperature and nucleus sampling produce candidates until <eoc>; 200 samples are drawn per problem.
Experiments
HumanEval contains 164 handcrafted Python problems with an average of 7.7 held-out tests per problem. MBPP contains 974 entry-level problems, split into 474 training and 500 test items. Metrics are Chen et al.’s unbiased pass@1, pass@10, and pass@100. Baselines include CodeX, CodeGen, AlphaCode, GPT-Neo, GPT-J, and InCoder. Decoding settings vary with k; missing libraries may be automatically imported during re-evaluation.
Results
HumanEval scores are 17.07/24.05/34.55% for 317M and 23.78/35.36/51.24% for 2.6B. The smaller model beats CodeX 300M on pass@1 and pass@10; the larger beats CodeX 2.5B on pass@1. On MBPP, 2.6B achieves 23.00/43.60/59.64%, versus InCoder 6.7B’s 19.40% pass@1. The advantage is concentrated at low k, while longer-context models dominate high-k sampling.
Applications
The system can support docstring-to-function completion in IDEs, educational programming assistants, contest-solution drafting, and boilerplate generation. Deployment requires static analysis, sandbox execution, dependency resolution, and CI testing. Since the model currently targets English and Python functions, it is better suited to localized assistance than autonomous repository-wide software engineering.
Limitations & Outlook
Docstrings vary in quality and may not specify algorithms precisely. Isolating functions removes imports, state, configuration, and cross-file contracts; automatic imports make evaluation more permissive than production execution. The short context limits long programs and descriptions, while the paper provides limited systematic ablations of objective weights. Future systems should combine repository retrieval, execution-guided repair, multilingual data, longer context, and PanGu-Coder-FT with CI feedback.
Plain Language Accessible to non-experts
Imagine a kitchen training a chef to cook from customer requests. In the first course, the chef reads millions of recipes and kitchen notes, learning common ingredients, steps, and styles. This resembles PanGu-Coder’s first training stage on Python code. In the second course, each lesson pairs a customer’s request with one finished dish. The chef practices turning the request and dish name into the missing cooking steps, while sometimes reconstructing hidden words in the request.
The final test is not whether the recipe looks familiar. Judges taste the dish using secret checks. HumanEval has 164 programming tasks, and MBPP has 974. PanGu-Coder’s 2.6B version succeeds on the first attempt for 23.78% of HumanEval tasks and 23.00% of MBPP tasks. It does this with a smaller kitchen and shorter workbench than several competitors.
But the chef usually works on one dish, not an entire restaurant. If tools, ingredients, or earlier preparations are missing, the result may fail. Very long orders can also exceed the workbench. So the model is a useful kitchen assistant, not yet a fully independent head chef.
ELI14 Explained like you're 14
Think of PanGu-Coder as a coding robot for a game. You tell it, “Write a function that finds the greatest common divisor,” give it the function name, and it tries to fill in the missing code. First, the robot studies huge numbers of Python files from GitHub, like learning the rules and tricks of a game. Then it practices a more specific challenge: reading a description and writing the function that matches it.
There are two robot sizes. On HumanEval, a set of 164 coding challenges, the 317M robot gets the first try right 17.07% of the time. The 2.6B robot gets 23.78%. On MBPP, it scores 23.00% on the first try and beats a much larger InCoder model. The researchers also ask for 200 possible answers per problem, because one clever answer may be hidden among many attempts.
The robot predicts code one piece at a time, rather like autocomplete. But it can only keep about 1,024 tokens in view. A giant problem may make it forget the beginning. It can also forget to bring in a needed library, so the researchers sometimes add missing imports during testing.
Why is this exciting? It shows that smarter practice materials can matter as much as making a robot bigger. The next upgrade would let it read a whole project, run its code, see errors, and repair itself. That would feel much more like having a real coding teammate!
Glossary
Causal Language Modeling (CLM)
CLM predicts the next token from all preceding tokens. Technically, it maximizes the autoregressive likelihood of a sequence.
It trains general Python knowledge in Stage 1 and predicts code in Stage 2 through CODE-CLM.
Masked Language Modeling (MLM)
MLM replaces selected tokens with masks or random alternatives and asks the model to recover them. It emphasizes contextual representation learning.
Docstr-MLM is applied only to the natural-language description subsequence.
Query Layer
An additional attention layer that uses the embedding of the next position as a query over final Transformer representations. It helps determine the next generated token.
PanGu-Coder inherits this component from PanGu-α.
pass@k
The estimated probability that at least one of k generated candidates passes all hidden tests. The paper uses Chen et al.’s unbiased estimator.
HumanEval and MBPP report pass@1, pass@10, and pass@100.
Abstract Syntax Tree (AST)
A tree representation of a program’s grammatical structure. It exposes functions, docstrings, signatures, and bodies without relying only on text patterns.
AST parsing filters files and extracts training functions.
Nucleus Sampling
A decoding method that samples from the smallest token set whose cumulative probability reaches p. Temperature separately controls distribution sharpness.
The paper tunes both parameters for different pass@k settings.
Open Questions Unanswered questions from this research
- 1 Can function-level training transfer reliably to whole repositories? Current examples omit cross-file dependencies, state, configuration, and version constraints that automatic imports cannot reproduce.
- 2 What are the optimal weights among CODE-CLM, Docstr-MLM, and Docstr-MCLM? The paper compares objective combinations but does not provide a broad, language- and length-controlled ablation.
- 3 Does hidden-test success correlate with secure and maintainable software? Future evaluation should include vulnerabilities, licensing, resource use, and adversarial inputs.
Applications
Immediate Applications
IDE function completion
A developer supplies an English docstring and Python signature; PanGu-Coder proposes the body, which is then checked by static analysis, sandbox execution, and unit tests. It is suitable for boilerplate and common algorithms, but should remain subject to review.
Programming education
Teachers and students can request alternative implementations, test cases, or starter solutions. HumanEval and MBPP results indicate useful entry-level capability, while execution-based checking is essential because plausible-looking code can still be wrong.
Long-term Vision
Verified software agents
Combined with retrieval, compilation, CI tests, and error feedback, the model could generate functions, execute them, and iteratively repair failures. Realization requires repository-level context, stronger dependency modeling, security isolation, and longer-term evaluation.
Abstract
We present PanGu-Coder, a pretrained decoder-only language model adopting the PanGu-Alpha architecture for text-to-code generation, i.e. the synthesis of programming language solutions given a natural language problem description. We train PanGu-Coder using a two-stage strategy: the first stage employs Causal Language Modelling (CLM) to pre-train on raw programming language data, while the second stage uses a combination of Causal Language Modelling and Masked Language Modelling (MLM) training objectives that focus on the downstream task of text-to-code generation and train on loosely curated pairs of natural language program definitions and code functions. Finally, we discuss PanGu-Coder-FT, which is fine-tuned on a combination of competitive programming problems and code with continuous integration tests. We evaluate PanGu-Coder with a focus on whether it generates functionally correct programs and demonstrate that it achieves equivalent or better performance than similarly sized models, such as CodeX, while attending a smaller context window and training on less data.