SemRep: Generative Code Representation Learning with Code Transformations
SemRep employs generative code representation with semantic-preserving transformations, improving correctness by 6.9%, performance by 1.1×, and robustness in code transformation tasks.
Key Findings
Methodology
SemRep integrates generative code representation learning via reinforcement learning (GRPO), training models to produce semantically equivalent code variants as intermediate representations. During training, the model explores diverse syntax while maintaining semantic invariance, guided by test-based rewards. In inference, it alternates between generating semantically equivalent code and instruction-specific transformations through iterative evolutionary search, enabling exploration of multiple optimization pathways. This approach combines explicit structural representations with exploration mechanisms, significantly enhancing the accuracy, generalization, and robustness of code transformations. Experiments demonstrate a 6.9% increase in correctness and a 1.1× performance boost over baselines, with improved cross-hardware transfer and robustness to semantic-preserving code perturbations.
Key Results
- On GPU kernel optimization, SemRep achieves 87.4% correctness, outperforming traditional finetuned models by 6.9%, with a 1.1× speedup, and improves cross-hardware correctness by 13.9%.
- In general code editing benchmarks, it reaches a Pass@1 of 53.7%, surpassing baseline models at 50.9%. When combined with evolutionary search, it uncovers optimizations that models 685 times larger fail to find, while reducing inference compute by 25%.
- Ablation studies confirm that explicit semantic representation learning and iterative exploration contribute significantly to these improvements, validating the effectiveness of the proposed framework.
Significance
This work addresses the core challenge of disentangling code semantics from syntactic variations, overcoming the limitations of implicit representations in large language models. By explicitly generating semantically equivalent code, SemRep enhances interpretability, generalization, and robustness, enabling more reliable automation in code optimization, scientific computing, and software maintenance. Its integration of structured intermediate representations with exploration mechanisms paves the way for scalable, explainable, and high-performance code transformation systems, advancing both academic research and industrial applications in automated software engineering.
Technical Contribution
SemRep introduces a novel framework combining explicit semantic-preserving code generation with reinforcement learning and evolutionary search. Its key innovations include the use of a generative mid-training task to produce diverse, semantically equivalent code variants, and a multi-stage inference process that iteratively refines code through exploration and instruction-guided transformation. This approach departs from traditional end-to-end models by explicitly modeling code semantics, enabling better generalization and robustness. The framework also leverages verifiable rewards based on test execution, ensuring correctness and facilitating reinforcement learning. These contributions collectively expand the toolkit for scalable, interpretable, and high-quality code transformation models.
Novelty
This is the first work to embed explicit, semantics-preserving code generation as a core intermediate step within a reinforcement learning framework, combined with iterative evolutionary search. Unlike prior models relying solely on implicit latent representations or static abstractions, SemRep explicitly constructs interpretable code variants, enabling more effective exploration and optimization. Its integration of generative learning, structured intermediate representations, and evolutionary algorithms represents a significant advancement in automated code transformation research, setting a new paradigm for robustness and scalability.
Limitations
- Despite improvements, the model may struggle with highly complex or ambiguous code semantics, especially in cases with insufficient test coverage or intricate logic, leading to potential correctness issues.
- Training and inference require substantial computational resources due to reinforcement learning and iterative search, limiting scalability in resource-constrained environments.
- Transferability to hardware-specific low-level optimizations remains limited; future work should incorporate hardware-aware learning to improve cross-platform performance.
Future Work
Future directions include integrating hardware-aware optimization strategies, reducing training costs via self-supervised learning, and extending the framework to multi-language and multi-task scenarios. Exploring unsupervised approaches for semantic preservation and scaling to larger codebases will further enhance practical applicability. Additionally, incorporating user feedback and explainability modules could improve usability and trust in automated code transformation tools.
AI Executive Summary
Code transformation plays a vital role in software engineering, enabling tasks like optimization, refactoring, and maintenance. Traditional methods often rely on static abstractions or end-to-end learning, which struggle to balance interpretability, robustness, and exploration. SemRep introduces a novel framework that leverages generative code representation learning guided by semantic-preserving transformations. By training models to produce diverse, semantically equivalent code variants as intermediate representations, SemRep enhances the model’s understanding of code semantics. During inference, it employs iterative evolutionary search to explore multiple transformation pathways, guided by explicit correctness and instruction adherence metrics.
This approach addresses key limitations of prior work, such as lack of interpretability and poor generalization, by explicitly modeling code semantics and enabling robust exploration. Experimental results on GPU kernel optimization and general code editing tasks demonstrate a 6.9% improvement in correctness, a 1.1× performance boost, and significant gains in cross-hardware transfer and robustness to code perturbations. Notably, SemRep can discover optimizations that much larger models fail to find, while reducing inference costs by 25%. These findings highlight its potential to revolutionize automated code optimization, scientific computing, and software maintenance.
The framework’s core innovations include the integration of reinforcement learning with explicit semantic code generation, multi-stage iterative inference, and exploration-driven optimization. This combination enables scalable, interpretable, and high-performance code transformation systems. Despite its strengths, challenges remain in handling highly complex semantics and reducing computational costs. Future work aims to incorporate hardware-aware learning, unsupervised training, and multi-language support, pushing the boundaries of automated software engineering. Overall, SemRep offers a promising new paradigm for intelligent, reliable, and efficient code transformation, with broad implications for academia and industry alike.
Deep Analysis
Background
近年来，深度学习在自然语言处理和代码生成领域取得突破，代表性模型如OpenAI的Codex、Google的CodeBERT等推动了自动代码理解与生成的发展。传统方法依赖静态抽象结构（如抽象语法树、控制流图）以确保语义一致性，但在复杂变换和自然语言指令引导下表现不足。端到端学习模型虽具一定灵活性，但隐式表示难以解释，泛化能力有限。近年来，研究者开始探索显式语义表示结合强化学习的方法，旨在提升模型对代码语义的理解和变换能力，为自动化软件工程提供新思路。
Core Problem
现有方法多依赖端到端训练，隐式表示缺乏可解释性，难以应对多样化变换和复杂逻辑。硬编码抽象虽保证语义一致，但缺乏灵活性，难以适应自然语言指令和多目标优化。如何构建既结构化又可解释的中间表示，兼顾多样性、鲁棒性和效率，成为核心难题。特别是在需要高精度和跨平台迁移的场景中，模型的泛化能力和鲁棒性亟待提升。
Innovation
SemRep提出将语义保持变换作为中间表示，结合强化学习（GRPO）进行生成式训练，显著增强模型对代码语义的理解。其创新点包括：• 生成语义等价代码作为中间步骤，提升模型的可解释性和探索能力；• 多轮演化搜索结合显式中间表示，增强优化空间的多样性；• 采用强化学习奖励机制，确保生成代码的语义一致性和正确性。这些创新突破了传统端到端模型的局限，为自动化代码变换提供了新思路。
Methodology
- �� 训练阶段：模型通过强化学习（GRPO）生成语义等价的代码变体，利用测试用例验证语义一致性，鼓励多样化探索；• 变换阶段：在指令引导下，模型利用生成的中间代码进行多轮演化搜索，逐步优化代码性能和结构；• 训练目标：最大化语义保持和指令符合的奖励，确保变换的正确性和多样性；• 结合结构化表示和探索机制，模型在保持功能的同时，生成多样化的变体以供后续优化。
Experiments
在GPU核优化和通用代码编辑两个任务上，使用KernelBench和EditBench数据集，评估模型的正确率、速度提升和鲁棒性。模型参数从7B到32B，训练采用RL和演化搜索结合策略。通过ablation验证生成式表示和探索机制的贡献，结果显示SemRep在正确率上提升6.9%，GPU核优化中性能提升1.1倍，跨硬件迁移正确率提升13.9%。
Results
SemRep在GPU核优化中正确率达87.4%，比微调模型提升6.9%，性能提升1.1倍，跨硬件迁移正确率提升13.9%。在代码编辑任务中，Pass@1达53.7%，优于基线50.9%。结合演化搜索，发现参数大685倍模型未能找到的优化方案，推理计算减少25%。这些数据验证了其探索能力和优越性能。
Applications
该技术适用于自动代码优化、GPU核设计、软件重构和维护等场景。开发者可利用SemRep自动生成高效、鲁棒的代码变体，减少人工调优时间。未来结合硬件感知和多模态信息，有望实现跨平台、跨语言的智能代码变换，推动软件工程智能化。
Limitations & Outlook
模型在极端逻辑复杂或语义偏离场景下仍可能失效，训练成本高，依赖大量测试用例验证。对硬件特定优化迁移能力有限，未来需结合硬件感知和无监督学习进行改进。
Plain Language Accessible to non-experts
想象你在厨房做菜，手里有一份食谱（代码），但你想让菜更快或更好吃。传统方法就像照着菜谱硬做，可能效果不理想。SemRep像厨师会根据原始菜谱，尝试不同的做法（变体），确保味道一样（语义保持），同时改进做法（优化性能）。它会不断试验不同的做法，找到最优的方案。这样，厨师不仅能做出好菜，还能学会多种做法，未来遇到不同食材（硬件）也能灵活应对。这种方法让厨房变得更智能，菜肴更美味，效率更高。
ELI14 Explained like you're 14
想象你在学校里做科学实验，你有一个实验步骤（代码），但你想让实验更快或更准确。以前，你只会照着步骤做，效果不一定好。现在，有个聪明的助手（SemRep），它会帮你试出不同的步骤变体，确保结果一样（语义保持），但用的时间更少或效果更好。它会不断尝试不同的方法，找到最棒的方案。这样，你的实验不仅变得更快，还能学会多种做法，未来遇到不同的材料（硬件）也能应付自如。这个助手让科学变得更聪明、更高效。
Abstract
Code transformation is a foundational capability in the software development process, where its effectiveness relies on constructing a high-quality code representation to characterize the input code semantics and guide the transformation. Existing approaches treat code transformation as an end-to-end learning task, leaving the construction of the representation needed for semantic reasoning implicit in model weights or relying on rigid compiler-level abstractions. We present SemRep, a framework that improves code transformation through generative code representation learning. Our key insight is to employ the semantics-preserving transformations as the intermediate representation, which serves as both a generative mid-training task and the guidance for subsequent instruction-specific code transformations. Across general code editing and optimization tasks (e.g., GPU kernel optimization), SemRep outperforms the extensively finetuned baselines with strictly the same training budget by 6.9% in correctness, 1.1x in performance, 13.9% in generalization, and 6.7% in robustness. With the improved exploration of diverse code transformations, SemRep is particularly amenable to evolutionary search. Combined with an evolutionary coding agent, SemRep finds optimizations that 685B larger-weight baselines fail to discover while achieving the same performance with 25% less inference compute.