PBT-Bench: Benchmarking AI Agents on Property-Based Testing
PBT-Bench evaluates AI's property-based testing ability using 100 curated problems with 365 semantic bugs across 40 Python libraries, revealing significant model performance gaps.
Key Findings
Methodology
This study constructs 100 property testing problems from 40 real Python libraries, injecting 365 semantic bugs across three difficulty levels. Each problem is paired with a precise Hypothesis @given strategy, requiring models to read documentation, identify invariants, and construct input strategies to trigger bugs. Two prompting regimes—open-ended and structured—are used to evaluate 8 contemporary large models through multiple runs. Metrics include bug recall, problem coverage, and full recall, providing a comprehensive assessment of models’ PBT capabilities.
Key Results
- Models under PBT-guided prompts achieved bug recall from 42.1% to 83.4%, outperforming open prompts (31.4% to 76.7%). Mid-capability models benefited over 20 percentage points from structured prompts, while the strongest models showed limited or negative gains, indicating that prompt structure can sometimes interfere with model behaviors.
- Different architectures exhibit distinct strengths and weaknesses across bug difficulty levels, with no single model closing all gaps. Ensemble approaches combining multiple models reached 99.5% overall problem coverage, highlighting the potential of model fusion.
- The evaluation reveals persistent model-specific weaknesses, especially on complex bugs, emphasizing the need for diverse architectures and training strategies to improve robustness in semantic bug detection.
Significance
This work fills a critical gap by providing a standardized, real-world benchmark for assessing AI's ability to perform property-based testing, a core skill involving understanding API invariants and constructing targeted input distributions. It advances the understanding of model capabilities in semantic reasoning and bug exposure, with implications for automated testing, software reliability, and AI interpretability. The benchmark’s open release fosters reproducibility and further research, promoting progress toward autonomous, reliable software testing agents.
Technical Contribution
The paper introduces PBT-Bench, a novel benchmark combining curated semantic bugs with explicit Hypothesis strategies, enabling precise evaluation of models’ invariant inference and input strategy construction. It innovates by stratifying bugs into difficulty levels, employing multi-model ensemble analysis, and automating evaluation via containerized F→P harnesses. These contributions distinguish it from existing patch or test generation benchmarks, emphasizing the conceptual skill of translating documentation into universal invariants.
Novelty
This is the first comprehensive benchmark explicitly targeting the core PBT skill—deriving invariants and constructing input strategies—using real, human-verified bugs. Unlike prior benchmarks focused on patch synthesis or test generation from minimal examples, PBT-Bench emphasizes semantic reasoning grounded in documentation, revealing nuanced model strengths and weaknesses across complex bug types.
Limitations
- The injected bugs, while carefully curated, may not encompass all real-world defect complexities, potentially limiting ecological validity.
- Evaluation depends on prompt design and hardware environment, which may influence results; future work should explore adaptive prompting and broader model architectures.
- Despite rigorous verification, some bugs may still be detectable by manual inspection or external effects, indicating room for further refinement.
Future Work
Future directions include expanding the library set to cover more domains and languages, developing adaptive input strategies, and integrating reinforcement learning to improve invariant inference. Additionally, exploring model training techniques that enhance semantic understanding and robustness will be critical. The benchmark can also serve as a foundation for developing explainability tools and automated bug explanation systems, ultimately advancing AI’s role in reliable software engineering.
AI Executive Summary
In the realm of software engineering, automating bug detection remains a formidable challenge. Traditional testing relies on concrete input-output pairs, which fall short when specifications are abstract, such as API invariants that must hold across diverse inputs. Large language models (LLMs) have demonstrated remarkable code understanding, yet their capabilities in property-based testing—deriving universal invariants and constructing precise input strategies—are not well understood.
PBT-Bench introduces a rigorous, real-world benchmark designed to evaluate these skills. By curating 100 property testing problems across 40 Python libraries, each embedded with multiple semantic bugs, the benchmark assesses whether models can read documentation, infer invariants, and generate Hypothesis strategies to trigger violations. These problems span three difficulty levels, from simple boundary bugs to complex cross-function protocol violations, providing a nuanced challenge.
Evaluation involves eight state-of-the-art models tested under two prompting regimes: a general, open-ended prompt and a structured, property-focused prompt. Multiple runs reveal that structured prompts significantly boost mid-capability models’ bug detection rates—by over 20 percentage points—yet have limited or even adverse effects on the strongest models. This indicates that prompt design influences model behavior differently depending on architecture and training.
Results show a wide performance gap across models and bugs, with no single model closing all challenges. Combining multiple models through ensemble strategies achieves near-complete coverage, emphasizing the importance of diversity in model architectures. The findings highlight the persistent difficulty of complex semantic bugs and the potential of collaborative model approaches.
Overall, PBT-Bench provides a vital tool for advancing AI's role in automated, semantic bug detection. Its open-source release encourages community engagement, fostering further research into invariant inference, input strategy construction, and robust AI-driven testing. The benchmark sets a new standard for evaluating AI capabilities in understanding and reasoning about software invariants, promising to accelerate progress toward autonomous software quality assurance.
Deep Dive
Abstract
Existing code benchmarks measure whether an agent can produce any test that reproduces a known bug, or whether it can produce a patch that fixes a described issue. Neither isolates the distinct skill of property-based testing: deriving a semantic invariant from documentation, and then constructing an input-generation strategy precise enough to make a random search reveal the violation. We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries. Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them; the agent must read the library's documentation, identify the relevant invariant, and specify a Hypothesis @given strategy that concentrates mass in the trigger region. Bugs are stratified across three difficulty levels (L1-L3) spanning single-constraint boundary bugs to stateful, cross-function protocol violations. We evaluate eight contemporary LLMs under two prompting regimes (open-ended baseline vs. explicit Hypothesis scaffolding) for three independent runs per configuration. Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%. Hypothesis scaffolding lifts mid-capability models by over 20 percentage points, but yields smaller gains for the strongest models, with two exceptions showing degradation, suggesting the structured prompt can interfere with certain model behaviours rather than complementing them. The hardest bugs prove model-specific: different architectures fail on different problems, leaving persistent gaps that no single model closes. We release the benchmark, harness, and full evaluation corpus to support downstream work on documentation-grounded semantic reasoning.