Small effect size leads to reproducibility failure in resting-state fMRI studies
2018
Thousands of papers using resting-state functional magnetic resonance imaging (RS-fMRI) have been published on brain disorders. Results in each paper may have survived correction for multiple comparison. However, since there have been no robust results from large scale meta-analysis, we do not know how many of published results are truly positives. The present meta-analytic work included 60 original studies, with 57 studies (4 datasets, 2266 participants) that used a
between-group designand 3 studies (1 dataset, 107 participants) that employed a within-group design. To evaluate the
effect sizeof brain disorders, a very large neuroimaging dataset ranging from neurological to psychiatric disorders together with healthy individuals have been analyzed. Parkinson9s disease off
levodopa(PD-off) included 687 participants from 15 studies. PD on
levodopa(PD-on) included 261 participants from 9 studies. Autism spectrum disorder (ASD) included 958 participants from 27 studies. The meta-analyses of a metric named
amplitudeof
low frequency fluctuation(ALFF) showed that the
effect size(Hedges9g) was 0.19 - 0.39 for the 4 datasets using
between-group designand 0.46 for the dataset using within-group design. The
effect sizeof PD-off, PD-on and ASD were 0.23, 0.39, and 0.19, respectively. Using the meta-analysis results as the robust results, the
between-group designresults of each study showed high false negative rates (median 99%), high
false discovery rates(median 86%), and low accuracy (median 1%), regardless of whether stringent or liberal multiple comparison correction was used. The findings were similar for 4 RS-fMRI metrics including ALFF, regional homogeneity, and degree centrality, as well as for another widely used RS-fMRI metric namely seed-based functional connectivity. These observations suggest that multiple comparison correction does not control for false discoveries across multiple studies when the
effect sizesare relatively small. Meta-analysis on un-thresholded t-maps is critical for the recovery of ground truth. We recommend that to achieve high reproducibility through meta-analysis, the neuroimaging research field should share raw data or, at minimum, provide un-thresholded statistical images.
Keywords:
-
Correction
-
Source
-
Cite
-
Save
39
References
10
Citations
NaN
KQI