Abstract: In supervised learning tasks, it is crucial for any algorithm to make accurate predictions on future data. As a Bayesian version of the gradient boosting algorithm, Bayesian additive regression trees (BART) have great potential to achieve high prediction accuracy. As far as we know, however, BART has not received as much attention as random forests and boosting. Thus, a comprehensive overview of BART is first presented to facilitate its understanding. Considering that BART may suffer from over-fitting in high-dimensional situations, one novel technique called RS-BART is developed to enhance its performance. Through first sorting all the variables with their relative importance, some low- or medium-dimensional BART models are trained with important variables. The predictions produced by these BART models are then integrated into the final result. By conducting experiments with some simulated and real data, RS-BART is demonstrated to perform better than or competitively with some state-of-the-art techniques including random forests, boosting and BART. Thus, RS-BART can be deemed as a competitive tool to solve real prediction tasks, especially high-dimensional but sparse ones.

Key words: ensemble learning, Bayesian additive regression tree, prediction accuracy, random forest, Gibbs sampling