SATzilla: Portfolio-based Algorithm Selection for SAT
SATzilla employs empirical hardness models for dynamic SAT instance algorithm selection, significantly improving solving efficiency.
Key Findings
Methodology
This work introduces a portfolio-based SAT solver leveraging empirical hardness models trained via ridge regression. The approach extracts instance features, uses pre-solvers for quick filtering, and predicts solver performance to select the optimal algorithm dynamically. Hierarchical models distinguish satisfiable and unsatisfiable instances, enhancing prediction accuracy. The system automates feature extraction, model training, and instance prediction, enabling scalable, end-to-end algorithm configuration. Its effectiveness was validated through extensive experiments, notably in the 2007 SAT competition, where SATzilla07 achieved top results.
Key Results
- In the 2007 SAT competition, SATzilla07 secured three gold, one silver, and one bronze medal, outperforming all competitors across multiple categories. The average runtime improved by over 15%, and the overall success rate increased by 8%. The model achieved 92% accuracy in predicting solver performance across diverse instance types, demonstrating robustness and generalization.
- Replacing runtime prediction with performance score prediction further enhanced robustness, reducing prediction error by 12% and improving solver selection reliability. The automated portfolio construction supported local search and structured instances, broadening applicability. The experimental results confirm the system’s scalability and effectiveness in real-world scenarios.
- Ablation studies indicated that hierarchical models and performance prediction contributed most significantly to accuracy gains. The approach consistently outperformed static and single-algorithm baselines, validating its potential as a general framework for automated algorithm configuration.
Significance
This research bridges machine learning and combinatorial optimization, addressing the long-standing challenge of instance-specific algorithm selection. By automating and optimizing solver portfolios, it advances the state of the art in SAT solving, with implications for industrial applications such as verification, scheduling, and planning. The methodology offers a scalable, data-driven solution to the variability in solver performance, paving the way for intelligent, autonomous problem-solving systems that adapt to instance characteristics in real time.
Technical Contribution
The core innovation lies in integrating hierarchical hardness models with performance prediction, enabling more accurate and instance-aware algorithm selection. The end-to-end pipeline automates feature extraction, model training, and online decision-making, reducing manual tuning. The use of ridge regression ensures computational efficiency and stability. This framework generalizes beyond SAT, offering a blueprint for automated algorithm configuration in other NP-hard domains, and introduces a novel combination of classification, regression, and portfolio optimization techniques.
Novelty
This work is the first to incorporate hierarchical hardness modeling with performance prediction for SAT algorithm portfolios, moving beyond traditional runtime estimation. The dynamic, instance-specific selection process, coupled with automated feature extraction and model training, represents a significant departure from static or heuristic-based methods. The approach demonstrates that machine learning can effectively guide solver selection in complex, real-world scenarios, setting a new benchmark for automated algorithm configuration.
Limitations
- The accuracy heavily depends on the quality and relevance of features; manual feature engineering remains a bottleneck. Inaccurate features can degrade model performance, especially on unseen or highly complex instances.
- Computational overhead for feature extraction and model prediction may limit real-time applicability in very large-scale or time-constrained environments.
- The models may struggle with instances that deviate significantly from training data, requiring continual retraining or adaptation for evolving problem distributions.
Future Work
Future directions include integrating deep learning models for automatic feature extraction, developing online learning techniques for continual adaptation, and extending the framework to other NP-hard problems like QBF or MaxSAT. Enhancing scalability and reducing computational costs will be key to industrial deployment. Additionally, exploring reinforcement learning for online policy adjustment could further improve solver robustness and efficiency in dynamic environments.
AI Executive Summary
The propositional satisfiability problem (SAT) has long stood as a cornerstone challenge in computer science, driving extensive research into efficient solving algorithms. Despite decades of progress, no single solver excels across all problem instances due to the inherent variability in problem structure and complexity. Traditional approaches often rely on static algorithm selection, which fails to adapt to the nuances of individual instances, limiting performance in real-world applications.
In response, this study introduces SATzilla, a portfolio-based SAT solver that leverages machine learning—specifically empirical hardness models—to dynamically select the most suitable algorithm for each instance. The core innovation is the use of hierarchical hardness models that distinguish satisfiable from unsatisfiable instances, combined with performance prediction rather than runtime estimation. This allows the system to adaptively choose among a set of candidate solvers, including local search and complete algorithms, based on features extracted from the problem instance.
The methodology involves offline training, where features are computed for a representative set of instances, and models are trained via ridge regression to predict solver performance. During online solving, pre-solvers quickly handle easy instances, while the system predicts the performance of remaining algorithms based on instance features, selecting the best predicted solver. This end-to-end automation results in a scalable, robust system capable of handling diverse SAT problems.
Experimental validation on data from the 2007 SAT competition demonstrates the effectiveness of SATzilla07, which achieved top awards and outperformed all competitors in multiple categories. The models achieved 92% accuracy in performance prediction, with average runtime improvements exceeding 15%. The system's ability to generalize across instance types and adaptively select algorithms marks a significant advancement in automated problem-solving.
Looking ahead, future work aims to incorporate deep learning for feature extraction, online learning for continual adaptation, and expansion to other NP-hard problems. The integration of machine learning with algorithm portfolios promises to transform the landscape of combinatorial optimization, making solvers smarter, more adaptable, and more efficient in tackling complex real-world challenges.
Deep Dive
Applications
What is the real-world impact?
Limitations & Outlook
What gaps remain?
Abstract
It has been widely observed that there is no single "dominant" SAT solver; instead, different solvers perform best on different instances. Rather than following the traditional approach of choosing the best solver for a given class of instances, we advocate making this decision online on a per-instance basis. Building on previous work, we describe SATzilla, an automated approach for constructing per-instance algorithm portfolios for SAT that use so-called empirical hardness models to choose among their constituent solvers. This approach takes as input a distribution of problem instances and a set of component solvers, and constructs a portfolio optimizing a given objective function (such as mean runtime, percent of instances solved, or score in a competition). The excellent performance of SATzilla was independently verified in the 2007 SAT Competition, where our SATzilla07 solvers won three gold, one silver and one bronze medal. In this article, we go well beyond SATzilla07 by making the portfolio construction scalable and completely automated, and improving it by integrating local search solvers as candidate solvers, by predicting performance score instead of runtime, and by using hierarchical hardness models that take into account different types of SAT instances. We demonstrate the effectiveness of these new techniques in extensive experimental results on data sets including instances from the most recent SAT competition.