When LLM Decompilers Recompile More and Preserve Less
Decompile-Diverge detects behavioral divergence in LLM decompilers, revealing a 13% divergence rate.
Key Findings
Methodology
The paper introduces Decompile-Diverge, which synthesizes drivers and generates fuzzing corpora to detect behavioral divergence in LLM decompilers. This method does not rely on fixed tests and can automatically generate inputs to compare the behavior of decompiled and original code.
Key Results
- Across eight systems in nine configurations, candidates that passed all original tests still showed a 4.9% overall divergence rate on the generated input corpus, with up to 13% for a single system.
- On 300 GitHub library functions and 287 CVE functions, recompilability increased from 75% to 90%, but behavioral agreement dropped from 74% to 62%.
- Up to one-tenth of disclosed vulnerabilities exhibited Crash Absence in the output.
Significance
This study highlights the risk of hidden vulnerabilities during recompilation by LLM decompilers, emphasizing the importance of behavioral consistency. Decompile-Diverge provides a new tool for security auditing and vulnerability analysis, filling gaps in existing test suites.
Technical Contribution
The technical contribution lies in proposing a behavioral comparison method that does not rely on fixed tests, dynamically generating inputs to detect behavioral changes in decompiled code. This offers a new perspective for evaluating LLM decompilers, revealing failure modes not captured by traditional tools.
Novelty
Decompile-Diverge is the first tool to compare decompilation behavior without relying on fixed tests, automatically generating inputs and detecting behavioral divergence, offering significant innovation over existing methods.
Limitations
- The method cannot be applied to types like multi-dimensional arrays and function pointers, limiting its applicability.
- Recompilation rates are low for certain project-specific types, affecting the detection of CVE functions.
Future Work
Future research could extend Decompile-Diverge's applicability to more data types and develop more efficient input generation algorithms to improve detection accuracy.
AI Executive Summary
Decompilation is the process of recovering high-level source code from compiled machine code, widely used in security auditing and vulnerability analysis. Traditional decompilers like Ghidra and Hex-Rays often leave some analyses unresolved, resulting in pseudocode that cannot compile or execute. Recently, LLM-based decompilers have shown significant improvements in recompilability and executability, generating clean, idiomatic C code. However, these metrics can be misleading, as the code may behave differently on other legitimate inputs, potentially hiding disclosed vulnerabilities.
To address this issue, the paper introduces the Decompile-Diverge method. This method synthesizes drivers and generates fuzzing corpora to detect behavioral divergence between decompiled and original code. Experimental results show that across eight systems in nine configurations, candidates that passed all original tests still showed a 4.9% overall divergence rate on the generated input corpus, with up to 13% for a single system.
This finding is significant for both academia and industry, revealing the risk of hidden vulnerabilities during recompilation by LLM decompilers and emphasizing the importance of behavioral consistency. Decompile-Diverge provides a new tool for security auditing and vulnerability analysis, filling gaps in existing test suites. Future research could extend the method's applicability to more data types and develop more efficient input generation algorithms to improve detection accuracy.
Deep Analysis
Background
Decompilation is a foundational task in security and software engineering, supporting critical tasks like security auditing, vulnerability triage, and legacy software maintenance. Traditional decompilers like Ghidra and Hex-Rays often leave some analyses unresolved, resulting in pseudocode that cannot compile or execute. Recently, LLM-based decompilers have shown significant improvements in recompilability and executability.
Core Problem
Existing recompilability and executability metrics can be misleading, as the code may behave differently on other legitimate inputs, potentially hiding disclosed vulnerabilities. This issue is not captured by existing test suites.
Innovation
The Decompile-Diverge method synthesizes drivers and generates fuzzing corpora to detect behavioral divergence between decompiled and original code. This method does not rely on fixed tests and can automatically generate inputs to compare the behavior of decompiled and original code.
Methodology
- �� Synthesize drivers to generate inputs
- �� Use AFL++ fuzzing to generate corpora
- �� Compare the behavior of decompiled and original code
- �� Detect crashes, hangs, and behavioral changes
Experiments
Experiments were conducted on eight systems in nine configurations, using 300 GitHub library functions and 287 CVE functions. By synthesizing drivers and generating fuzzing corpora, the method detects behavioral divergence between decompiled and original code.
Results
Despite passing all original tests, candidates showed a 4.9% overall divergence rate on the generated input corpus, with up to 13% for a single system. Up to one-tenth of disclosed vulnerabilities exhibited Crash Absence in the output.
Applications
Decompile-Diverge can be used for security auditing and vulnerability analysis, helping detect hidden vulnerabilities during decompilation. This method provides a new perspective for evaluating LLM decompilers.
Limitations & Outlook
The method cannot be applied to types like multi-dimensional arrays and function pointers, limiting its applicability. Recompilation rates are low for certain project-specific types, affecting the detection of CVE functions.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Traditional decompilers are like a recipe book with some unclear steps, leaving you to guess. LLM decompilers are like a smart chef who fills in those steps but sometimes adds ingredients that shouldn't be there. Decompile-Diverge is like a strict food critic, tasting each dish to ensure it matches the original recipe. This way, even if the chef's dish looks good, it can still spot the discrepancies.
ELI14 Explained like you're 14
Imagine you're playing a complex video game. Traditional decompilers are like a game guide with some blurry parts, leaving you to figure things out. LLM decompilers are like a game master who helps you speed through but sometimes takes the wrong path. Decompile-Diverge is like a strict referee, checking every move to ensure you're on the right track. This way, even if the master helps you win, it can still spot the mistakes.
Glossary
Decompilation
The process of recovering high-level source code from machine code, widely used in security and software engineering.
Used for vulnerability detection and malware analysis.
Large Language Model (LLM)
A deep learning-based model capable of generating natural language text.
Used to generate idiomatic C code.
Behavioral Divergence
The phenomenon where decompiled code behaves differently from the original code under the same inputs.
Used to evaluate the accuracy of decompilers.
Fuzzing
An automated testing technique that generates random inputs to detect software vulnerabilities.
Used to generate input corpora.
Crash Absence
The disappearance of a disclosed vulnerability's crash phenomenon in decompiled code.
Used to evaluate the security of decompilers.
Open Questions Unanswered questions from this research
- 1 How to improve Decompile-Diverge's applicability to multi-dimensional arrays and function pointers?
- 2 How to increase recompilation rates for project-specific types?
Applications
Immediate Applications
Security Auditing
Helps security researchers detect hidden vulnerabilities during decompilation, improving code security.
Long-term Vision
Automated Vulnerability Analysis
By improving Decompile-Diverge, automate the detection and fixing of security vulnerabilities in code.
Abstract
Decompilation recovers high-level source from compiled machine code and serves as a foundation for security tasks such as vulnerability detection and malware analysis. Traditional decompilers like Ghidra and Hex-Rays expose whatever they cannot resolve as visible placeholders and often emit pseudocode that will not compile or execute; LLM-based decompilers produce clean, idiomatic C and are now judged almost entirely by recompilability and re-executability: whether the output builds and passes its shipped input/output tests. We show that these metrics can reward the wrong path: a function may recompile and pass every shipped test yet diverge on other legitimate inputs, and a disclosed vulnerability may disappear from the recompiled code with no visible trace of the crash. Neither failure is caught by existing suites. To address this gap, we propose Decompile-Diverge, a behavioral comparison oracle not relying on fixed or hand-crafted tests: for each function it synthesizes a driver, grows a fuzzing corpus from the reference, and reruns the decompiled code on the same inputs to detect changes in the function's behavior. Across eight systems in nine configurations on established LLM decompilation corpora, candidates that pass every shipped test still diverge from the original on our input corpus: 4.9% overall, and as many as 13% for a single system. On 300 real GitHub library functions and 287 CVE-grounded functions, recompilability and behavioral agreement can come apart: the strongest refinement LLM lifts Ghidra's build rate from 75% to 90%, while its Matched rate falls from 74% to 62%; on disclosed vulnerabilities, up to one tenth exhibit Crash Absence in its output. Source-level analysis traces this divergence to introduced fields, types, callees, and guards that replace the visible unknowns traditional tools leave behind.