Bootstrap aggregating improves the generalizability of Connectome Predictive Modelling

It is a long-standing goal of neuroimaging to produce reliable generalized models of brain behavior relationships. More recently data driven predicative models have become popular. Overfitting is a common problem with statistical models, which impedes model generalization. Cross validation (CV) is often used to give more balanced estimates of performance. However, CV does not provide guidance on how best to apply the models generated out-of-sample. As a solution, this study proposes an ensemble learning method, in this case bootstrap aggregating, or bagging, encompassing both model parameter estimation and feature selection. Here we investigate the use of bagging when generating predictive models of fluid intelligence (fIQ) using functional connectivity (FC). We take advantage of two large openly available datasets, the Human Connectome Project (HCP), and the Philadelphia Neurodevelopmental Cohort (PNC). We generate bagged and non-bagged models of fIQ in the HCP. Over various test-train splits, these models are evaluated in sample, on left out HCP data, and out-of-sample, on PNC data. We find that in sample, a non-bagged model performs best, however out-of-sample the bagged models perform best. We also find that feature selection can vary substantially within-sample. A more considered approach to feature selection, alongside data driven predictive modeling, is needed to improve cross sample performance of FC based brain behavior models.