Improving the Predictive Performance of Bootstrap Aggregating by Dirichlet Resampling

Random forests are a machine learning method that improves predictive performance by integrating multiple decision trees. Since Breiman introduced bagging in 1996, random forests have become popular for their strong generalization ability and ease of use. However, high inter-tree correlation limits their performance. Researchers have tried random feature selection and extremely randomized trees to reduce correlation, but with limited success.

The core problem of random forests is high inter-tree correlation, limiting their generalization ability. Traditional solutions like random feature selection and extremely randomized trees reduce correlation to some extent but cannot fully solve the problem. How to further reduce inter-tree correlation without weakening individual trees is a pressing issue.

The core innovation of this paper is the introduction of a Dirichlet resampling mechanism, which flexibly controls sample weight distribution by adjusting the concentration parameter α, thereby reducing inter-tree correlation. Compared to traditional methods, this innovation provides new theoretical guarantees and high flexibility in engineering implementation.

• �� Dirichlet-Multinomial Bagging Random Forest (DM): Generates each tree's sample bag via a Dirichlet-multinomial distribution, adjusting α to control sample overlap.

• �� Dirichlet-Weighted Random Forest (DW): Assigns Dirichlet weights to each sample, retaining all training points' support.

• �� Provides a theoretical criterion for choosing α, ensuring that DM and DW are indistinguishable from standard random forests at a certain resolution.

Experiments used 15 benchmark datasets from UCI and OpenML, comparing DM and DW with traditional random forests, extremely randomized trees, and other baselines. Main evaluation metrics include classification accuracy, log-loss, and AUROC. α selection was optimized through random search.

On the adult dataset, DM achieved a classification accuracy of 85.61%, significantly outperforming baseline methods. DW showed statistically significant advantages on multiple datasets, especially in handling class imbalance. Experimental results indicate that DM and DW improve predictive performance in most cases without additional computation time.

DM and DW methods have broad application potential in large-scale datasets and class imbalance issues, especially suitable for scenarios requiring high accuracy and robustness, such as financial risk assessment and medical diagnosis.

Although DM and DW perform excellently on multiple datasets, their performance improvement is not significant on some datasets, possibly limited by the complexity of data features. Further optimization of α parameter selection is needed to adapt to different data characteristics and application scenarios.