EquivSVA: A Formally Verified Dataset of Behavioral Assertions Across Equivalent RTL Implementations
EquivSVA offers a formally verified dataset with 120 behavior families and 480 RTL implementations.
Key Findings
Methodology
EquivSVA dataset is organized around behavior families, each containing four structurally distinct but behaviorally equivalent RTL implementations, shared interface-level gold properties, three controlled mutants, and formal validation evidence. A 17-job validation suite ensures the correctness of each family.
Key Results
- Of 293 generated properties, 93 are formally sound, with 14 families showing variability across equivalent implementations.
- The Qwen2.5-Coder-7B-Instruct model exhibits mutation sensitivity across 24 test families.
- The dataset supports studying assertion generation robustness without altering intended functionality.
Significance
EquivSVA supports controlled studies of assertion generation systems' robustness by providing multiple equivalent implementations, addressing long-standing issues of assertion dependency on specific RTL implementations, significant for both academia and industry.
Technical Contribution
EquivSVA introduces a new dataset organization method, integrating multiple equivalent RTL implementations, shared properties, and mutants into a single unit, supporting robustness research. It offers new engineering possibilities for formal verification.
Novelty
EquivSVA is the first to use behavioral equivalence as a core dataset organizing principle, rather than an auxiliary perturbation applied post-construction.
Limitations
- The dataset is programmatically generated, possibly not reflecting the full complexity of industrial code.
- It only covers control logic and small state machines, not complex structures like large datapaths.
Future Work
Future work could expand to more complex behavior categories, such as large datapaths and protocol stacks, and explore longer temporal properties.
AI Executive Summary
EquivSVA is an innovative dataset focused on behavioral equivalence in formal verification. Existing assertion generation methods often rely on specific RTL implementations, but EquivSVA addresses this by providing 120 behavior families, each with four structurally distinct but behaviorally equivalent RTL implementations. The dataset's validation process includes 17 tasks to ensure each family's correctness and robustness.
In experiments, the Qwen2.5-Coder-7B-Instruct model showed different mutation sensitivities across 24 test families, indicating that different implementations might affect the soundness of generated assertions. This way, EquivSVA provides a controlled experimental environment for studying assertion generation robustness.
The release of EquivSVA offers a powerful tool for academia and industry to research and improve formal verification methods. It supports evaluating existing methods and provides a foundation for developing new assertion generation technologies. Future work could expand to more complex behavior categories, enhancing the dataset's application value.
Deep Analysis
Background
Formal verification plays a crucial role in digital hardware design, especially in ensuring design intent and safety properties. With large language models applied to generate SystemVerilog assertions from natural language, existing datasets support large-scale training and formal evaluation. However, whether generated assertions depend on specific RTL implementations remains a challenge.
Core Problem
The core problem is ensuring that generated assertions apply not only to specific RTL implementations but also consistently across equivalent implementations. This is crucial for verifying system robustness, as RTL can have multiple implementations.
Innovation
EquivSVA introduces a new research framework by organizing datasets around behavioral equivalence. Each behavior family contains four equivalent RTL implementations, shared gold properties, and controlled mutants, supporting in-depth studies of assertion generation robustness.
Methodology
- �� Each behavior family starts from a machine-readable specification.
- �� Contains four structurally distinct RTL implementations ensuring behavioral equivalence.
- �� Provides shared interface-level gold properties and three controlled mutants.
- �� Ensures family correctness through a 17-job validation suite.
Experiments
The experimental design includes evaluating the Qwen2.5-Coder-7B-Instruct model using 24 test families and 96 RTL inputs. The study examines the soundness and mutation sensitivity of generated assertions across different implementations.
Results
Results show that of 293 generated properties, 93 are formally sound. Fourteen families exhibit variability in soundness across equivalent implementations, indicating that implementation style may affect assertion generation.
Applications
EquivSVA can be used to evaluate the robustness of assertion generation systems, particularly across multiple equivalent implementations. It provides a controlled experimental environment for formal verification research.
Limitations & Outlook
The dataset's generation method may not fully reflect the complexity of industrial code. Additionally, it currently covers only control logic and small state machines, with potential for expansion to more complex structures.
Plain Language Accessible to non-experts
Imagine you're in a kitchen preparing a big meal. Each dish has different recipes, but the final taste should be the same. EquivSVA is like a cookbook, where each recipe has different steps, but the final dish tastes the same. This way, we can test if different chefs (assertion generation systems) can produce the same delicious dish (consistent assertions) under different recipes.
ELI14 Explained like you're 14
Imagine you're playing a game where each level has a different map, but the goal is the same. EquivSVA is like these levels, each with different paths but the same goal. We want to see if different players (assertion generation systems) can complete the same task (generate consistent assertions) on different maps.
Glossary
RTL (Register Transfer Level)
RTL is an abstraction level in hardware design describing signal transfers between clock cycles.
Used to describe equivalent hardware implementations.
SVA (SystemVerilog Assertions)
SVA is used to verify temporal and safety properties of hardware designs.
Generated assertions are used to test robustness across equivalent implementations.
Behavior Family
A set of structurally distinct but behaviorally equivalent RTL implementations.
Core organizational unit of the dataset.
Gold Properties
Shared interface-level properties used to verify behavioral consistency.
Ensures behavioral equivalence across different implementations.
Mutants
Controlled behavior changes used to test the soundness of assertions.
Used to verify the robustness of generated assertions.
Open Questions Unanswered questions from this research
- 1 How to apply EquivSVA's organizing principles to more complex structures?
- 2 How to expand the dataset to cover a broader range of behaviors?
Applications
Immediate Applications
Formal Verification Evaluation
Researchers can use EquivSVA to evaluate the robustness of assertion generation systems across multiple equivalent implementations.
Educational Tool
EquivSVA can serve as a teaching tool to help students understand behavioral equivalence in formal verification.
Long-term Vision
Industrial Application
EquivSVA can be used to develop more robust assertion generation tools, enhancing verification efficiency in industrial hardware design.
Abstract
Large language models are increasingly used to generate SystemVerilog Assertions from natural-language specifica- tions and register-transfer-level designs. Existing datasets and benchmarks support important goals such as large- scale training, formal evaluation, specification-to-assertion generation, and mutation-based testing. A complemen- tary need is to study whether a generated assertion cap- tures externally observable behavior or depends on inci- dental details of one RTL implementation. We present EquivSVA, a formally verified dataset organized around behavior families. Each family contains four structurally distinct RTL implementations of the same externally ob- servable behavior, shared interface-level gold properties, three controlled mutants, and formal-validation evidence. EquivSVA contains 120 behavior families across 12 cat- egories, 480 reference RTL implementations, 914 gold properties, and 360 mutants. Every final family passes a fixed 17-job validation suite covering RTL equivalence, gold-property proofs, property reachability, mutant dis- tinguishability, and gold-property checks on mutants. We also provide fixed family-safe train, development, and test splits. As a small demonstration of the analyses en- abled by the dataset, we evaluate the publicly released, Apache-2.0-licensed Qwen2.5-Coder-7B-Instruct model on the held-out test split. Of 293 interface-only generated properties, 93 are formally sound, and the number of sound properties varies across equivalent implementations for 14 of 24 test families. These results illustrate how behavior-family organization can support controlled stud- ies of assertion-generation robustness without requiring changes in intended functionality. The dataset, generators, validation scripts, and case-study artifacts are publicly released at https://github.com/aditigupta96/EquivSVA.