VRank: Enhancing Verilog Code Generation from Large Language Models via Self-Consistency
VRank leverages self-consistency and chain-of-thought reasoning to improve Verilog code generation accuracy by 10.5% on average, using multiple candidate sampling and clustering.
Key Findings
Methodology
VRank employs multiple sampling from large language models (LLMs) to generate diverse Verilog candidates and a testbench. It runs simulations to obtain output vectors, then clusters candidates based on identical outputs, reflecting functional equivalence. Clusters are scored using a modified Minimum Bayes Risk (MBR) approach, measuring output consistency. Top clusters are further refined via Chain-of-Thought (CoT) reasoning, where LLMs analyze discrepancies and infer correct signals. This integrated approach automates candidate selection, significantly improving functional correctness without human intervention. The core algorithms include probabilistic sampling, output-based clustering, inconsistency scoring, and multi-round CoT analysis.
Key Results
- On the VerilogEval-Human benchmark, VRank improves pass@1 accuracy by an average of 10.5% across models such as GPT-4, Llama-3, and CodeV-Qwen, with the highest gain of 19.6%. Experiments show performance gains stabilize with as few as 20 samples, achieving 90% of the maximum improvement. The method effectively reduces errors, especially in low-performing models, and enhances robustness across diverse design tasks.
- Across models, VRank consistently outperforms baseline random sampling, with notable improvements in models like CodeV-Qwen (up to 19.6%). The integration of output consistency clustering and CoT reasoning leads to more accurate candidate selection, especially in complex scenarios involving application and connection descriptions, where accuracy increases significantly.
- Chain-of-Thought reasoning further refines top candidate selection by analyzing output discrepancies and inferring correct signals, boosting pass@1 accuracy in challenging tasks. The combined approach demonstrates strong generalization, stability, and scalability, making it suitable for industrial deployment.
Significance
This work addresses the longstanding challenge of low functional correctness in LLM-generated HDL code, offering a fully automatic, high-precision solution. By systematically leveraging probabilistic sampling, output-based clustering, and logical reasoning, VRank reduces reliance on manual verification, accelerating hardware design workflows. The approach bridges the gap between AI capabilities and practical engineering needs, paving the way for scalable, automated chip development and verification. Its broad applicability across models and design complexities highlights its potential to transform hardware automation, reduce costs, and improve reliability in semiconductor manufacturing.
Technical Contribution
The primary innovation lies in combining probabilistic candidate generation with output-based clustering and a novel scoring mechanism inspired by MBR. The use of Chain-of-Thought reasoning to analyze output discrepancies introduces a new layer of logical validation, enhancing selection accuracy. This integrated framework systematically filters out low-quality candidates, providing a robust pipeline for HDL code synthesis. The methodology extends existing self-consistency principles from machine translation to hardware code generation, establishing a new paradigm for automated verification and synthesis with theoretical guarantees of improved correctness.
Novelty
This research is the first to systematically incorporate self-consistency clustering and Chain-of-Thought reasoning into HDL code generation. Unlike prior methods relying on multiple retries or manual validation, VRank automates candidate filtering through output similarity and logical analysis, significantly reducing human effort. Its innovative use of probabilistic sampling combined with multi-round reasoning sets a new standard for AI-driven hardware design automation, offering a scalable, reliable, and fully automated solution.
Limitations
- The effectiveness depends on the quality of the generated testbenches and the diversity of candidate samples. Poorly designed testbenches may lead to incorrect clustering or candidate selection.
- Computational costs increase with the number of samples and reasoning rounds, potentially limiting real-time applications in large-scale industrial settings.
- The approach may struggle with extremely complex or novel hardware designs outside the training distribution, necessitating further robustness improvements.
Future Work
Future directions include integrating formal verification techniques to further validate generated code, optimizing sampling and reasoning algorithms for efficiency, and extending the framework to other hardware description languages like VHDL. Additionally, combining reinforcement learning to guide candidate sampling and reasoning could further enhance accuracy and scalability. Exploring multi-modal inputs, such as schematic diagrams or behavioral descriptions, may also broaden applicability. Ultimately, the goal is to develop a fully autonomous hardware design pipeline that seamlessly integrates code synthesis, verification, and deployment.
AI Executive Summary
As chip complexity surges, automating the generation of high-quality Verilog code becomes increasingly critical. Traditional methods rely heavily on manual inspection or multiple retries, which are inefficient and costly. This paper introduces VRank, an innovative framework that harnesses the probabilistic nature of large language models (LLMs) to generate multiple candidate codes, then employs output-based clustering and scoring to identify the most promising solutions automatically.
VRank’s core process involves generating diverse Verilog candidates, running them on LLM-created testbenches, and clustering based on identical simulation outputs. This output consistency serves as a proxy for functional correctness. To refine candidate selection, the framework applies a modified Minimum Bayes Risk (MBR) score, quantifying the similarity among outputs. The top clusters are then analyzed using Chain-of-Thought (CoT) reasoning, where the LLM evaluates discrepancies and infers the correct signals, further improving accuracy.
Experimental results on the VerilogEval-Human benchmark demonstrate that VRank significantly outperforms baseline random sampling, with an average pass@1 accuracy increase of 10.5%, reaching up to 19.6% in some models. The performance gains are stable across different sample sizes, with 20 samples already capturing most improvements. The approach proves effective across various models, including GPT-4, Llama-3, and CodeV-Qwen, showing its robustness and generalizability.
This work marks a substantial step toward fully automated hardware design workflows, reducing reliance on human verification and accelerating chip development cycles. Its innovative use of self-consistency and logical reasoning opens new avenues for AI-assisted hardware synthesis. Future research will focus on integrating formal verification, optimizing computational efficiency, and extending to other HDL languages, aiming for a scalable, reliable, fully autonomous hardware design ecosystem.
Deep Dive
Abstract
Large Language Models (LLMs) have demonstrated promising capabilities in generating Verilog code from module specifications. To improve the quality of such generated Verilog codes, previous methods require either time-consuming manual inspection or generation of multiple Verilog codes, from which the one with the highest quality is selected with manually designed testbenches. To enhance the generation efficiency while maintaining the quality of the generated codes, we propose VRank, an automatic framework that generates Verilog codes with LLMs. In our framework, multiple code candidates are generated with LLMs by leveraging their probabilistic nature. Afterwards, we group Verilog code candidates into clusters based on identical outputs when tested against the same testbench, which is also generated by LLMs. Clusters are ranked based on the consistency they show on testbench. To determine the best candidate, Chain-of-Thought is further applied to select the best candidate from the top-ranked clusters. By systematically analyzing diverse outputs of generated codes, VRank reduces errors and enhances the overall quality of the generated Verilog code. Experimental results on the VerilogEval-Human benchmark demonstrate a significant 10.5% average increase in functional correctness (passl1) across multiple LLMs, demonstrating VRank's effectiveness in improving the accuracy of automated hardware description language generation for complex design tasks.