MultiPruneEnsemble: Integrating Multi-Base Decision Tree Prunings via Gradient-Based Weighting

Abstract

The decision tree is widely used due to its interpretability. However, it usually leads to overfitting, which limits the accuracy of their prediction. Surprisingly, research on integrating these two approaches remains somewhat limited. In this paper, we propose a new framework, MultiPruneEnsemble, which integrates the diversity of pruning with the flexibility of gradient-based optimization. We train multiple independent base trees and apply several advanced pruning techniques to each, aiming to get a set of structurally different but functionally similar subtrees. Then we integrate these pruned models into an ensemble and use gradient descent to learn their weights so that the contribution of each tree is adapted to the data. To validate our algorithm, we ran it on multiple benchmark data 
sets and compared it with single-pruning baselines, simple averaging ensembles, and other standard tree models. Across all test datasets, our strategy yielded consistent gains in precision, F1 score, and AUC, with performance remaining stable on both large-scale and small-scale tasks. This indicates that integrating multiple decision tree pruning by gradient-based weighting is a feasible way to improve the accuracy and robustness of learning. We release our code and experiment logs to support reproducibility (see the Code Availability statement).