CERT: Continual Pre-Training on Sketches for Library-Oriented Code Generation
CERT uses sketch-then-generation continual pre-training, raising PandasEval pass@1 to 28.42%.
Key Findings
Methodology
CERT decomposes library-oriented generation into two stages. A sketcher anonymizes user-defined constants or names and predicts an abstract code sketch; a generator receives the sketch concatenated with the original context and restores concrete code. Both models are continually pre-trained from PYCODEGPT or CODEGEN on unlabeled library code. At inference, the sketcher samples 200 candidates and selects the mode, while training remains standard autoregressive language modeling.
Key Results
- On PandasEval, PYCODEGPT-CERT reaches 28.42% pass@1 versus 12.75% for PYCODEGPT, an absolute gain of 15.67 points. Its pass@10 and pass@100 are 48.04% and 60.96%, showing that sketches improve both first-choice reliability and candidate coverage.
- On NumpyEval, CODEGEN-CERT obtains 32.00% pass@1, 49.45% pass@10, and 67.82% pass@100, improving over CODEGEN by 12.69, 8.56, and 7.24 points. The 110M-parameter PYCODEGPT-CERT also exceeds 175B-parameter GPT-3 on PandasEval, 28.42% versus 12.97% pass@1.
- Anonymizing only user-defined constants is best: CERT-N and CERT-NC reach 23.66% and 19.07% pass@1 on PandasEval. The sketcher’s exact-match sketch accuracy exceeds baselines by 15.20 points on PandasEval and 14.21 points on NumpyEval.
Significance
The paper addresses a persistent practical bottleneck: third-party libraries are numerous, while high-quality natural-language/code pairs are expensive to annotate for every API usage pattern. CERT shows that unlabeled code can provide structural supervision through recurring library idioms. PandasEval and NumpyEval isolate library competence from the mostly standalone-function setting of HumanEval, creating executable benchmarks closer to data-analysis practice. The results suggest that domain-specific continual pre-training plus structured intermediate representations can make relatively small models useful at substantially lower cost than giant general models.
Technical Contribution
The contribution is more than domain adaptation. CERT constructs two complementary self-supervised corpora. The sketcher corpus globally anonymizes source files, teaching the model to predict API-oriented structure; the generator corpus interleaves anonymized and original blocks, teaching detail recovery conditioned on a sketch. No paired text/code annotations or additional supervised loss are required. The paper also evaluates constant-only, name-only, and joint anonymization, exposing a concrete trade-off between abstraction and semantic information. This provides a reproducible recipe for turning raw library code into hierarchical training signals.
Novelty
Unlike Coarse-to-Fine and PLOTCODER, which generally use sketches during supervised fine-tuning with labeled pairs, CERT moves sketching into continual pre-training on unlabeled code. Its fundamental innovation is to create training targets through deterministic code transformation and to learn them from library-specific subcorpora. The work therefore treats repeated API usage patterns as an intermediate representation and a source of self-supervision.
Limitations
- Evaluation covers only Python’s Pandas and NumPy, with 101 problems each. Generalization to other languages, libraries, multi-library workflows, and larger software-engineering tasks remains unestablished.
- Each module is trained for 100K steps on extracted files, while inference uses 200 sketch samples and temperature selection. The paper does not fully quantify end-to-end training, latency, or energy costs.
- Predefined anonymization symbols may be brittle. Over-abstraction removes useful semantics: joint name-and-constant anonymization lowers performance, showing that a sketch can become too vague.
Future Work
Future work should extend CERT to more libraries, languages, cross-library calls, and repository-level tasks. Promising directions include automatically learning anonymization granularity, structure-aware block segmentation, retrieval augmentation, and execution-guided refinement. More complete reporting of compute cost is needed, as are static analysis, type checking, test generation, and feedback loops that allow the generator to revise a flawed sketch or implementation.
AI Executive Summary
Code-generation systems usually benefit from expensive natural-language/code pairs, yet the long tail of third-party libraries makes such annotation impractical. CERT starts from an observation: library snippets often repeat the same API-oriented structure while changing variables, constants, and names. The paper turns this regularity into supervision without labels and introduces PandasEval and NumpyEval, two executable benchmarks containing 101 problems each.
CERT has two components. The sketcher replaces user-defined elements with symbols such as string, number, and variable, then predicts an abstract sketch. The generator receives that sketch together with the original context and fills in concrete details. Training uses library-specific unlabeled files: anonymized files for the sketcher, and alternating sketch/original code blocks for the generator. Both models are continually pre-trained for 100K steps from PYCODEGPT or CODEGEN, rather than fine-tuned on paired examples.
The gains are substantial. PYCODEGPT-CERT reaches 28.42% pass@1 on PandasEval, versus 12.75% for PYCODEGPT, an absolute improvement of 15.67 points; its pass@10 and pass@100 are 48.04% and 60.96%. On NumpyEval, CODEGEN-CERT reaches 32.00% pass@1, 12.69 points above CODEGEN. A 110M-parameter CERT model also exceeds GPT-3’s 12.97% PandasEval pass@1. Yet the evidence is limited to two Python libraries, and sampling plus preprocessing adds cost. The broader message is clear: for library-oriented coding, planning reusable structure before instantiating details can convert raw code into effective self-supervision.
Deep Analysis
Background
Pre-training produced systems such as CodeGPT, CodeParrot, GPT-Neo, and Codex; Codex 12B reaches 28.81% pass@1 on HumanEval. However, HumanEval emphasizes standalone functions and underrepresents API-intensive data analysis. More than 40% of Python-tagged StackOverflow questions also carry at least one library tag, showing the practical importance of library use.
Core Problem
Given context x—comments, imports, function headers, and variables—the model must generate target code y containing library calls. The challenge is the scarcity of paired descriptions and implementations across a huge API space. Direct generation must simultaneously infer control structure, API composition, names, and constants, so one local error can invalidate the entire program.
Innovation
- ��CERT decomposes generation into sketcher and generator.
- ��Deterministic anonymization creates structural targets without annotation.
- ��Library-specific subcorpora focus learning on relevant API idioms.
- ��Interleaved sketch/original blocks simulate sketch-conditioned generation.
- ��PandasEval and NumpyEval provide execution-based library benchmarks rather than relying only on generic code tests.
Methodology
- ��Data: from PYCODEGPT’s 13.0M Python files, extract about 0.61M Pandas files and 2.62M NumPy files.
- ��Sketching: transform each file d into d̄; the default replaces user-defined constants, while CERT-N replaces names and CERT-NC replaces both.
- ��Sketcher: continue pre-training on d̄ for 100K steps; infer s=MS(x) using 200 candidates and majority selection.
- ��Generator: split d and d̄ into function/class-like blocks and create [d̄1,d1,…,d̄K,dK]; continue pre-training for 100K steps.
- ��Decoding: generate y=MG([s;x]); return a complete sketch directly, or use x alone when the sketch is empty.
Experiments
Problems originate from highly voted StackOverflow posts with accepted answers, then are manually polished and tested. Each benchmark has 101 tasks; function tasks generally have 20 tests, while non-function tasks have one. Baselines include PYCODEGPT, CODEGEN, their XL continual-pretraining variants, CodeT5, CodeGPT-Adapted, CodeClippy, and CodeParrot. Metrics are pass@1/10/100, estimated from 200 samples per task using Chen et al.’s unbiased formula. Training uses a 1024-token window, learning rate 5e-4, FP16, and 100K continual-pretraining steps.
Results
PYCODEGPT-CERT scores 28.42/48.04/60.96% on PandasEval, compared with 12.75/37.80/59.65% for PYCODEGPT. On NumpyEval, CODEGEN-CERT scores 32.00/49.45/67.82%, versus 19.31/40.89/60.58% for CODEGEN. CERT improves across groups with different API counts. CERTg, trained on general Python rather than library-specific files, reaches only 8.25% pass@1 on HumanEval, confirming the importance of targeted data. Constant-only anonymization is consistently strongest.
Applications
CERT fits data-analysis copilots, notebook completion, StackOverflow assistance, enterprise Python scripting, and API migration. A deployment needs target-library source, version-aware documentation, and executable tests or static checks. Its 110M-parameter configuration is attractive for local IDEs and constrained environments, but generated code should remain sandboxed and verified before execution.
Limitations & Outlook
The evidence is restricted to Pandas and NumPy, so effects on deep-learning frameworks, databases, other languages, and multi-library workflows are unknown. Predefined symbols and manual block segmentation may depend on Python style. Complex control flow, hidden data semantics, and repository-level dependencies can still cause failures. Two 100K-step models and 200-sample decoding add cost. Future work should combine retrieval, execution feedback, learned sketch granularity, and broader software-engineering evaluation.
Plain Language Accessible to non-experts
Imagine a kitchen receiving an order: “Use these ingredients to make fried rice.” A novice cook may guess everything at once—the pan, sequence, heat, ingredient names, and quantities—and one wrong detail can ruin the dish. CERT first assigns a menu planner, who ignores the exact bag of rice and writes only the reusable procedure: wash, heat, stir-fry, season. That abstract recipe is the sketcher’s output.
A second cook, the generator, receives both the recipe and the actual ingredients. It turns “season” into concrete operations using the right variables, numbers, and Pandas or NumPy functions. During training, the kitchen does not need customers to label every order. It collects many completed dishes, temporarily replaces specific ingredient names with generic labels, and teaches one worker to recognize procedures and the other to restore details.
This works because similar dishes share techniques. On PandasEval, CERT raises the small model’s first-correct-answer rate from 12.75% to 28.42%. But recipes cannot be made too vague: hiding every name and value removes useful information, and the paper finds that hiding constants alone works best. Real kitchens still taste-test every dish; generated code likewise needs execution tests and safety checks.
ELI14 Explained like you're 14
Imagine building a bridge in a game. The mission tells you where the bridge starts and ends, and what materials you have. A normal robot tries to guess the entire build immediately: every plank, number, label, and position. If one tiny detail is wrong, the bridge may collapse. CERT gives the robot a planning step first: draw the bridge shape, place the supports, and only then choose the exact materials.
It uses two robots. The Sketcher draws the plan, while the Generator turns the plan into working code. They learn by reading lots of existing Python programs, not by receiving thousands of teacher-made answer pairs. During practice, names and numbers are replaced by labels like “name” or “number,” so the first robot learns common structures. The second robot practices filling those labels with real details.
The result is impressive: on PandasEval, the original 110M-parameter model gets a correct first answer 12.75% of the time, while CERT reaches 28.42%. On NumPy tasks, CODEGEN-CERT reaches 32.00%. So the model first asks, “What kind of move is needed?” instead of randomly guessing every detail at once.
But it is not magic! The tests cover only Pandas and NumPy, and a plan that removes too much information can confuse the model. Generated programs still need to run safely, just like a bridge must pass a load test. Future versions could learn more libraries and repair code after seeing an error.
Glossary
Sketch
An abstracted code structure in which selected user-defined details are anonymized while API calls and organization remain. It represents a reusable library-usage pattern.
CERT predicts a sketch first and then conditions the generator on it.
Continual pre-training
Further autoregressive training of an existing language model on new domain data. Unlike supervised fine-tuning, it does not require labeled input/output pairs.
Both CERT modules are continually pre-trained for 100K steps.
Sketcher
The model that predicts an abstract version of the target code from context. Its output may be incomplete, anonymous, or already executable.
CERT samples 200 candidate sketches and selects the most frequent one.
Generator
A language model that produces concrete complete code from the original context plus a sketch.
It is trained on alternating anonymized and original code blocks.
pass@k
The estimated probability that at least one of k generated candidates is correct. CERT generates 200 candidates and uses an unbiased estimator.
PandasEval and NumpyEval report pass@1, pass@10, and pass@100.
PandasEval / NumpyEval
Execution-based benchmarks for Pandas and NumPy code generation. Each contains 101 problems derived from StackOverflow.
They measure functional library use rather than only textual similarity.
Open Questions Unanswered questions from this research
- 1 Why does the optimal anonymization granularity vary across libraries? The paper tests three hand-designed rules but does not learn abstraction levels from API semantics.
- 2 The causal link between sketch errors and final program errors is unclear. The generator sometimes tolerates an imperfect sketch, but the boundary of this robustness is not systematically measured.
- 3 Real repositories combine libraries, data, versions, and hidden state. How retrieval, types, execution feedback, and security constraints should be integrated remains open.
Applications
Immediate Applications
IDE data-analysis completion
A developer supplies comments, imports, and variables; CERT predicts a Pandas or NumPy structure and instantiates concrete code. Organizations can continue training on internal repositories and use unit tests to filter unsafe or non-executable suggestions.
StackOverflow answer assistance
The system can convert a user question into an API sketch, then generate an explanation and example. Retrieval of versioned documentation, sandbox execution, and human review are needed to control API drift and incorrect calls.
Long-term Vision
Verifiable multi-library coding agents
A future agent could plan a cross-library sketch, retrieve documentation, perform type checks, execute tests, and revise implementation iteratively. Extending this approach across languages could turn unlabeled repositories into actionable engineering knowledge.
Abstract
Code generation is a longstanding challenge, aiming to generate a code snippet based on a natural language description. Usually, expensive text-code paired data is essential for training a code generation model. Recently, thanks to the success of pre-training techniques, large language models are trained on large-scale unlabelled code corpora and perform well in code generation. In this paper, we investigate how to leverage an unlabelled code corpus to train a model for library-oriented code generation. Since it is a common practice for programmers to reuse third-party libraries, in which case the text-code paired data are harder to obtain due to the huge number of libraries. We observe that library-oriented code snippets are more likely to share similar code sketches. Hence, we present CERT with two steps: a sketcher generates the sketch, then a generator fills the details in the sketch. Both the sketcher and the generator are continually pre-trained upon a base model using unlabelled data. Furthermore, we craft two benchmarks named PandasEval and NumpyEval to evaluate library-oriented code generation. Experimental results demonstrate the impressive performance of CERT. For example, it surpasses the base model by an absolute 15.67% improvement in terms of pass@1 on PandasEval. Our work is available at https://github.com/microsoft/PyCodeGPT.