On the Robustness of LLMs' Internal Representation of Code Correctness

The evolution of transformer-based LLMs like GPT, BERT, and subsequent instruction-tuned models such as Codex, CodeLlama, and Qwen has significantly advanced automatic code generation. These models excel at completing code snippets, generating functions, and even entire modules, transforming software development workflows. However, their outputs often lack reliable correctness signals, as traditional metrics like confidence scores and test executions are either poorly calibrated or computationally expensive. Recent studies, including Ribeiro et al. (2022), demonstrated that internal hidden states encode high-level semantic properties, including truthfulness and code correctness. These insights suggest that internal representations could serve as proxies for correctness assessment, offering a faster, test-free evaluation method. Nonetheless, the robustness and generalizability of such internal signals remain underexplored, especially across different models, datasets, and configurations.

Despite promising initial results, the internal correctness signals extracted via linear probing are sensitive to multiple factors—extraction location, prompt framing, and construction method—leading to inconsistent performance across models and datasets. The signals tend to degrade in out-of-distribution scenarios, limiting practical deployment. Furthermore, the data used to build these signals, often generated from model failures or canonical solutions, may not fully capture the complexity of real-world errors. This variability hampers the development of reliable, model-agnostic internal correctness indicators, constraining their utility in automated verification, debugging, and safety-critical applications.

This paper introduces a comprehensive, systematic framework for analyzing how different configurations—construction methods (PCA vs. mean-difference), prompt framings (neutral, concept, function), and extraction locations (last token vs. response average)—affect the robustness of internal correctness signals. It innovates by integrating controlled mutation and refactoring techniques to generate diverse, behavior-preserving error samples, enabling rigorous testing of data impact. The study evaluates four instruction-tuned models across multiple datasets, providing a detailed variance analysis and revealing configuration sensitivities. This multi-faceted approach advances understanding of internal representation stability and guides the design of more robust extraction strategies.

• �� Use linear probing to contrast hidden states of correct and incorrect programs, constructing correctness directions. • Vary extraction location (last token vs. response average) and prompt framing (neutral, concept, function). • Implement PCA and mean-difference methods for direction construction, comparing their performance. • Generate controlled error pairs via mutation operators (arithmetic, variable, control flow) and validate with test suites. • Apply behavior-preserving refactors to produce confounded variants, testing data impact. • Conduct experiments on four models (Mistral, Qwen, OpenCoder, CodeLlama) and two benchmarks (HumanEval, BigCodeBench), using cross-validation to select optimal layers and configurations. • Measure performance via pairwise and selection accuracy, analyzing variance contributions.

• �� Datasets include HumanEval and BigCodeBench, covering diverse programming tasks and external library calls. • Use 10-fold cross-validation, ensuring robustness. • Compare configurations across extraction methods, prompt framings, and locations. • Generate error pairs through 20 mutation operators, validate failures with test suites, and create confounded variants via refactoring. • Evaluate performance metrics—pairwise accuracy and selection accuracy—across models and datasets, analyzing configuration effects and robustness. • Conduct ablation studies to assess the impact of data augmentation and configuration choices.

• �� No single configuration consistently outperforms others; performance varies with setup. • Mean-difference construction exhibits better generalization across models and datasets. • Extraction location and prompt framing significantly influence accuracy, with models performing worse out-of-distribution. • Data augmentation via mutants and refactors does not substantially improve signals, indicating data limitations. • The highest observed accuracy is 63%, highlighting the challenge of stable internal correctness signals. These results suggest that internal representations are sensitive and require more robust extraction strategies for practical use.

• �� Automated code review: internal signals can supplement traditional testing for quick error detection. • Candidate ranking: internal correctness scores can prioritize promising solutions. • Rapid verification: test-free assessment accelerates development cycles. • Educational tools: internal signals help visualize correctness reasoning, aiding learning and debugging.

• �� Current signals are insufficiently accurate for critical applications, with maximum around 63%. • Configuration sensitivity and dataset limitations hinder robustness, especially out-of-distribution. • Data augmentation strategies showed limited improvements, indicating the need for richer, more diverse data. • Model internal representations remain fragile, necessitating further research into more stable extraction methods and larger, more varied training datasets. Future work should explore multi-modal signals, adaptive extraction techniques, and larger-scale models to address these challenges.