Trust, but Validate the Instrument: Auditing AI-Generated RTL Verification Plans on Authored Security-Regression Proxies
SecTB-RTL framework audits AI-generated RTL verification plans, revealing mismatches in generation and execution rules.
Key Findings
Methodology
The SecTB-RTL framework covers 31 tasks and 124 hardware security regressions, using a deterministic non-AI baseline for comparison. Task-level studies ensure generated verification plans remain unchanged in the trusted production path. The framework is designed to ensure that each generated verification plan is evaluated under standard simulation, synthesis, and formal tools.
Key Results
- In the C1-R3 run, only 9 out of 1,860 calls passed production semantic validation, highlighting mismatches in generation and execution rules.
- The deterministic baseline killed 36, 75, and 78 mutants at increasing resource limits, demonstrating benchmark feasibility.
- Generated verification plans may fail to detect security regressions even with high structural coverage.
Significance
This study highlights the limitations of AI-generated RTL verification plans in hardware security, emphasizing the importance of consistency between generation and execution. It provides a methodological foundation for future AI-assisted hardware design, helping to identify potential weaknesses in generated verifiers.
Technical Contribution
The SecTB-RTL framework provides a systematic approach to auditing AI-generated RTL verification plans, emphasizing consistency between generation and execution rules. Through task-level studies, the framework ensures the integrity of verification plans in the trusted production path.
Novelty
This study is the first to systematically audit the execution validity of AI-generated RTL verification plans, highlighting the importance of consistency between generation and execution rules.
Limitations
- Generated verification plans may fail to detect security regressions even with high structural coverage.
- Production semantic validation passed only 9 calls, indicating mismatches in generation and execution rules.
Future Work
Future work will include improving consistency between generation and execution rules and developing more effective verification plan generation methods.
AI Executive Summary
In hardware design, AI-generated RTL verification plans can satisfy provider schemas but may fail at the boundary of trusted execution. The SecTB-RTL framework audits this consistency issue across 31 tasks and 124 hardware security regressions. The study found that while most responses were accepted by providers, only a few passed production semantic validation, revealing mismatches in generation and execution rules. This research emphasizes the importance of consistency between generation and execution in verification plans, providing a methodological foundation for future AI-assisted hardware design. Although the study reveals limitations in generated verification plans, it offers directions for improving verification plan generation methods.
Deep Analysis
Background
With the advancement of AI technology, its application in chip design is shifting from code completion to generating verification artifacts. AutoBench has demonstrated that large language models can generate self-checking HDL testbenches from design descriptions, but this introduces a trust calibration problem. While generated artifacts can parse, run, and cover much of the design, they may lack the stimulus or oracle needed to detect security property violations.
Core Problem
AI-generated RTL verification plans may fail to provide measurable security evidence when evaluated under standard simulation, synthesis, and formal tools, despite satisfying provider schemas.
Innovation
The SecTB-RTL framework ensures that each generated verification plan remains unchanged in the trusted production path through task-level studies. The framework is designed to ensure that generated verification plans can detect hidden CWE-specific regressions when evaluated under standard simulation, synthesis, and formal tools.
Methodology
- �� Use a deterministic non-AI baseline for comparison
- �� Task-level studies ensure generated verification plans remain unchanged in the trusted production path
- �� Evaluate generated verification plans under standard simulation, synthesis, and formal tools
Experiments
The experimental design includes 31 tasks and 124 hardware security regressions, using a deterministic non-AI baseline for comparison. In the experiments, generated verification plans are evaluated under standard simulation, synthesis, and formal tools to detect hidden CWE-specific regressions.
Results
The study found that while most responses were accepted by providers, only a few passed production semantic validation, revealing mismatches in generation and execution rules. The deterministic baseline killed 36, 75, and 78 mutants at increasing resource limits, demonstrating benchmark feasibility.
Applications
This study provides a methodological foundation for AI-assisted hardware design, helping to identify potential weaknesses in generated verifiers. It offers directions for improving verification plan generation methods.
Limitations & Outlook
Generated verification plans may fail to detect security regressions even with high structural coverage. Production semantic validation passed only 9 calls, indicating mismatches in generation and execution rules.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. An AI-generated RTL verification plan is like a recipe that tells you how to make a dish. You prepare all the ingredients according to the recipe and start cooking. However, despite the recipe looking perfect, you find that the final dish doesn't taste as expected. This is because some steps in the recipe didn't account for actual cooking conditions like heat and time. The SecTB-RTL framework is like a chef who checks if each step is executed correctly, ensuring the final dish meets the expected taste.
ELI14 Explained like you're 14
Imagine you're playing a game where you have to build a fortress. The AI-generated RTL verification plan is like a building guide in the game, telling you how to build the fortress step by step. You follow the guide but find that the final fortress isn't stable. This is because the guide didn't consider some important details like material choice and structural stability. The SecTB-RTL framework is like a game master who checks each step's execution, ensuring the final fortress is strong and durable.
Glossary
RTL (Register Transfer Level)
RTL is an abstraction level used to describe the behavior of digital circuits, commonly used in hardware design and verification.
In the paper, RTL is used to describe the behavior of hardware designs for verification purposes.
CWE (Common Weakness Enumeration)
CWE is a classification system for software and hardware security vulnerabilities, helping to identify and fix security issues.
In the paper, CWE is used to identify security regressions in hardware designs.
Mutation Testing
Mutation testing is a software testing technique that introduces faults to evaluate the effectiveness of test cases.
In the paper, mutation testing is used to evaluate the effectiveness of generated verification plans.
Verification Plan
A verification plan is a set of steps and tests used to verify the correctness of a hardware design.
In the paper, verification plans are generated by AI to detect security regressions in hardware designs.
Production Semantic Validation
Production semantic validation refers to verifying the effectiveness of generated verification plans in actual production environments.
In the paper, production semantic validation is used to assess whether generated verification plans meet production standards.
Open Questions Unanswered questions from this research
- 1 How to improve consistency between generation and execution rules to enhance verification plan effectiveness?
- 2 How to detect undetected security regressions even with high structural coverage?
Applications
Immediate Applications
Hardware Design Verification
The SecTB-RTL framework can be used to verify AI-generated RTL verification plans, ensuring their effectiveness in production environments.
Long-term Vision
AI-Assisted Hardware Design
By improving consistency between generation and execution rules, AI can more effectively assist in hardware design and verification.
Abstract
AI-generated RTL verification plans can satisfy a provider schema yet fail at the boundary to trusted execution. We present SecTB-RTL, an auditable framework covering 31 tasks and 124 authored hardware-security regressions. A deterministic non-AI baseline killed 36, 75, and 78 mutants at increasing resource limits. The first confirmatory run (C1-R2) failed before model execution because the provider rejected its response schema. After a schema-only repair made without viewing outcomes, a separately frozen follow-up run (C1-R3) completed 1,860 calls. The provider accepted 1,857 responses, but only nine passed the production semantic validator. The generation and execution rules did not match. We therefore preserve the run as an instrument-validation incident and report no prompt-effect estimate. This incident shows that provider or schema acceptance does not establish execution validity. Compilation and coverage are only diagnostics; the exact saved artifact must pass the full production path. A subsequent follow-up is excluded because it did not satisfy the preregistered evidence-completeness gate and is treated only as future work. We release the benchmark, failure-preserving contract, incident provenance, and governance controls needed to prevent infrastructure behavior from being misreported as model behavior.