CodeTaste: Can LLMs Generate Human-Level Code Refactorings?

The integration of AI into software engineering has evolved from simple code completion to complex automated repair and refactoring tools. Early models like CodeBERT and GPT-3 demonstrated capabilities in code understanding and generation, primarily in single-file, task-specific contexts. Recent large models such as Codex and GPT-4 have shown potential in multi-task scenarios, but their evaluation remains limited to syntactic correctness or small-scale benchmarks. Existing benchmarks like CodeRefactor and SWE-Refactor focus on single-file edits, lacking the complexity of real-world, multi-file, multi-language projects. As software systems grow in complexity, the need for models that can understand and perform semantic, preference-aligned refactorings across large codebases becomes critical. This work addresses this gap by creating a comprehensive, multi-file, multi-language benchmark that evaluates models' ability to recognize and implement developer preferences in complex refactoring tasks.

Current AI models excel at generating syntactically correct code but struggle with autonomous recognition of human developer preferences, especially in large, multi-file projects. Existing benchmarks do not adequately measure a model’s ability to infer and align with developer intent, limiting their practical utility. The core challenge lies in designing evaluation metrics and datasets that reflect real-world complexity, including semantic understanding, multi-file coherence, and preference modeling. Without such benchmarks, progress in creating truly autonomous, preference-aware coding agents remains constrained. Addressing this requires integrating static analysis, dataflow reasoning, and preference signals into a unified evaluation framework.

This paper introduces CodeTaste, a benchmark that combines large-scale open-source refactoring data with static analysis rules and dataflow reasoning to evaluate models' semantic understanding and preference alignment. Key innovations include:

1) Mining multi-file refactoring instances from GitHub, capturing real developer behavior.

2) Generating detailed task descriptions and static rules to evaluate semantic correctness.

3) Implementing a propose-then-implement framework that leverages multiple candidate plans and preference signals.

4) Supporting multi-language, multi-file, large-scale evaluation, surpassing prior single-file, syntactic benchmarks.

These innovations enable a more realistic assessment of models' capabilities in complex software engineering tasks, emphasizing preference understanding and long-term maintainability.

• �� Data collection: Mining 100 high-quality multi-file refactoring commits from GitHub repositories across six languages.
• �� Task generation: Using LLMs to create detailed refactoring instructions, including rationale and target patterns.
• �� Environment setup: Building reproducible, containerized environments with test suites and static analysis rules.
• �� Static rule creation: Automatically generating additive and reductive rules to capture desired and undesired patterns.
• �� Model inference: Running models within environments, generating patches via propose-then-implement, with multi-round reasoning and preference signals.
• �� Evaluation: Validating correctness via test suites and measuring preference alignment through static rule matching.
• �� Metrics: Alignment scores, functional correctness, rule coverage, and file/line modifications.
• �� Ablation studies: Testing the impact of multi-round inference, preference signals, and static rules on performance.

The evaluation involved five models, including GPT-5.2, Claude 4.5, and Qwen3, tested on 100 multi-file refactoring tasks. Metrics included test pass rate (>95%), preference alignment (up to 70%), and code modifications (average 91 files, 2605 lines). Experiments demonstrated that multi-round, preference-guided inference significantly improved alignment, with the best models reaching 20%+ in autonomous preference tasks. Cross-language performance was consistent, validating the benchmark's robustness. Ablation studies confirmed the importance of preference signals and iterative reasoning in enhancing model performance.

Models achieved up to 69.6% alignment in detailed instruction tasks, with high functional correctness. In autonomous preference tasks, alignment was below 8%, but multi-round inference and preference signals raised this to over 20%. Static analysis rules covered diverse semantic patterns, validating models' semantic understanding. The proposed framework effectively bridges the gap between detailed instructions and autonomous preference detection, highlighting the importance of iterative reasoning and preference modeling.

The benchmark can guide development of AI tools for automated code maintenance, refactoring, and debt reduction. It supports training models to autonomously recognize developer preferences, enabling more intelligent code suggestions and long-term software evolution. Industrial applications include automated refactoring in CI/CD pipelines, reducing manual effort and improving code quality. The framework also facilitates research on preference modeling, multi-task learning, and semantic understanding in large models.

Models still underperform in autonomous preference detection due to limited semantic reasoning and static rule expressiveness. Evaluation mainly uses open-source repositories, limiting industrial robustness validation. Static rules require manual creation, which may introduce biases. Future work should focus on integrating dynamic analysis, automatic rule learning, and real-world industrial datasets to improve robustness and applicability.