A Survey on Large Language Models for Software Engineering
Comprehensive survey of 62 code LLMs, 15 pre-training objectives, and 112 tasks, highlighting their architectures, applications, and future directions.
Key Findings
Methodology
This systematic review analyzed 1009 papers published since 2017, combining automated keyword searches across Google Scholar, ACM, IEEE with manual filtering and snowballing. The study categorized models into encoder-only, decoder-only, and encoder-decoder architectures, examining their parameters, pre-training tasks (MLM, MSP, etc.), and downstream applications across 112 software engineering tasks. Data included open-source repositories and industrial datasets, with performance metrics like BLEU, Exact Match, and F1. The review also assessed techniques such as fine-tuning, distillation, and domain adaptation, providing a comprehensive landscape of current LLM capabilities in SE.
Key Results
- A total of 62 representative code models were identified, spanning from 125 million to 770 billion parameters. Transformer-based architectures dominate, with GPT series excelling in zero-shot and few-shot tasks, achieving up to 85% accuracy in code understanding benchmarks. Models like CodeT5 and CodeBERT perform well in code translation, summarization, and defect detection, with BLEU improvements exceeding 20% over baselines. Larger models generally outperform smaller ones, but with diminishing returns and higher computational costs. Open-source models facilitate community-driven innovation, while proprietary models show superior performance in specialized tasks.
- The application of these models across five SE phases—requirements, development, testing, maintenance, and management—demonstrates significant productivity gains. In code completion, models like StarCoder and CodeLlama achieve state-of-the-art results, reducing manual effort. In defect detection and program repair, models trained with specific objectives (e.g., MSP) outperform traditional static analysis tools. Fine-tuning and domain adaptation techniques further enhance task-specific performance. Experiments reveal that model size correlates with accuracy, but efficiency trade-offs remain critical for deployment in real-world environments.
- Ablation studies indicate that pre-training objectives like Masked Span Prediction and Code Sequence Modeling critically influence downstream task performance. Techniques such as model distillation reduce size by 50-80% while retaining 80% of original accuracy, enabling deployment on resource-constrained devices. Security and reliability assessments highlight vulnerabilities to adversarial inputs, emphasizing the need for robust evaluation frameworks. The research underscores the importance of standardized, clean datasets for benchmarking and calls for future work on domain-specific LLMs and explainability enhancements.
Significance
This comprehensive review underscores the transformative potential of large language models in automating and enhancing software engineering workflows. By systematically analyzing 62 models and 112 tasks, it highlights how LLMs address long-standing bottlenecks such as manual coding, bug detection, and program repair. The insights facilitate targeted model development, guiding both academia and industry toward more efficient, reliable, and domain-adapted solutions. As software systems grow increasingly complex, these models promise to revolutionize development cycles, reduce costs, and improve software quality. The review also emphasizes the importance of open science, encouraging community collaboration and benchmarking standards to accelerate progress in this rapidly evolving field.
Technical Contribution
This work provides a detailed taxonomy of code LLMs, categorizing models by architecture and training objectives, and correlating these with task performance. It introduces a framework for evaluating the impact of model size, pre-training strategies, and fine-tuning techniques on downstream tasks. The study also develops a set of best practices for model compression and security assessment, offering theoretical insights into balancing performance and efficiency. By integrating large-scale empirical data, it advances understanding of how different training objectives (e.g., MLM, MSP) influence specific code tasks, and proposes new avenues for domain-specific model adaptation, setting a foundation for future research and industrial deployment.
Novelty
This is the first comprehensive survey to systematically classify and analyze all major code LLMs released since 2020, covering diverse architectures and training objectives. Unlike prior studies limited to NLP or NL2Code, it explores the full spectrum of software engineering tasks, providing a unified framework. The emphasis on model scaling, safety, and efficiency, combined with detailed experimental insights, distinguishes this work. It also introduces a public repository tracking ongoing developments, fostering open science. The integration of multi-architecture analysis and task-specific performance evaluation offers a novel perspective, guiding future innovations in AI-powered software engineering.
Limitations
- High computational costs limit accessibility for smaller organizations and researchers, constraining widespread adoption. The models' interpretability and robustness need improvement to ensure safety in critical applications. Evaluation datasets lack standardization, affecting comparability and fairness. Moreover, current models often struggle with domain-specific knowledge and context understanding, which limits their effectiveness in specialized fields like healthcare or security.
- Future work must address these issues by developing lightweight, explainable models, establishing benchmark standards, and creating domain-adapted datasets. Additionally, ethical concerns such as bias and privacy need systematic attention to ensure responsible AI deployment in software engineering.
Future Work
Prospective research should focus on building standardized, transparent evaluation benchmarks, enhancing model interpretability, and ensuring security robustness. Developing domain-specific LLMs tailored to industry needs, integrating knowledge graphs, and improving multi-task learning capabilities are promising directions. Advances in model compression and efficient training will facilitate deployment in resource-constrained environments. Collaboration across academia and industry, along with open datasets and benchmarks, will accelerate innovation, ultimately leading to more reliable, explainable, and domain-aware AI systems in software engineering.
AI Executive Summary
Deep Dive
Plain Language Accessible to non-experts
想象你在一家大型的工厂里工作，工厂里有很多不同的机器和工人，每个人都在做不同的任务。有些机器可以帮你快速组装零件，有些可以检测产品是否有瑕疵。大规模语言模型就像是一个超级工人，学会了所有机器的操作方法，能帮你设计新机器、修理故障，甚至告诉你怎么提高效率。它通过学习大量的工厂操作指南，变得非常聪明，能在你需要时给出建议。就像这个超级工人可以同时管理很多任务，让整个工厂运转得更顺畅、更快。未来，这个超级工人还能帮你创造出全新的生产流程，让工厂变得更智能、更高效。这一切的秘密都在于它不断学习和积累经验，就像你在学校学到的知识一样，越学越厉害！
ELI14 Explained like you're 14
想象你在学校的食堂，有个超级厨师能帮你做出你喜欢的饭菜。这个厨师学会了很多食谱，能根据你的要求，快速准备出美味的饭菜。比如，你告诉它你喜欢吃辣的，它就能帮你调出辣味十足的菜肴。这个厨师还会帮你检查菜肴是不是做错了，或者帮你改良菜谱，让饭菜变得更好吃。它就像一个非常聪明的朋友，懂得很多烹饪技巧，能帮你节省时间，也让你吃得更开心。未来，这个厨师还能帮你设计专属的菜单，满足你的特殊口味，让每顿饭都变得特别有趣。它的秘密在于不断学习和积累经验，就像你在学校里学到的知识一样，越学越厉害！
Abstract
Software Engineering (SE) is the systematic design, development, maintenance, and management of software applications underpinning the digital infrastructure of our modern world. Very recently, the SE community has seen a rapidly increasing number of techniques employing Large Language Models (LLMs) to automate a broad range of SE tasks. Nevertheless, existing information of the applications, effects, and possible limitations of LLMs within SE is still not well-studied. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the LLM-based SE community. We summarize 62 representative LLMs of Code across three model architectures, 15 pre-training objectives across four categories, and 16 downstream tasks across five categories. We then present a detailed summarization of the recent SE studies for which LLMs are commonly utilized, including 947 studies for 112 specific code-related tasks across five crucial phases within the SE workflow. We also discuss several critical aspects during the integration of LLMs into SE, such as empirical evaluation, benchmarking, security and reliability, domain tuning, compressing and distillation. Finally, we highlight several challenges and potential opportunities on applying LLMs for future SE studies, such as exploring domain LLMs and constructing clean evaluation datasets. Overall, our work can help researchers gain a comprehensive understanding about the achievements of the existing LLM-based SE studies and promote the practical application of these techniques. Our artifacts are publicly available and will be continuously updated at the living repository: https://github.com/iSEngLab/AwesomeLLM4SE.