Type I and Type II error concerns in fMRI research: re-balancing the scale.
Statistical thresholding (i.e. P-values) in fMRI research has become increasingly conservative over the past decade in an attempt to diminish Type I errors (i.e. false alarms) to a level traditionally allowed in behavioral science research. In this article, we examine the unintended negative consequences of this single-minded devotion to Type I errors: increased Type II errors (i.e. missing true effects), a bias toward studying large rather than small effects, a bias toward observing sensory and motor processes rather than complex cognitive and affective processes and deficient meta-analyses. Power analyses indicate that the reductions in acceptable P-values over time are producing dramatic increases in the Type II error rate. Moreover, the push for a mapwide false discovery rate (FDR) of 0.05 is based on the assumption that this is the FDR in most behavioral research; however, this is an inaccurate assessment of the conventions in actual behavioral research. We report simulations demonstrating that combined intensity and cluster size thresholds such as P < 0.005 with a 10 voxel extent produce a desirable balance between Types I and II error rates. This joint threshold produces high but acceptable Type II error rates and produces a FDR that is comparable to the effective FDR in typical behavioral science articles (while a 20 voxel extent threshold produces an actual FDR of 0.05 with relatively common imaging parameters). We recommend a greater focus on replication and meta-analysis rather than emphasizing single studies as the unit of analysis for establishing scientific truth. From this perspective, Type I errors are self-erasing because they will not replicate, thus allowing for more lenient thresholding to avoid Type II errors.
Lieberman
Matthew D
MD
Department of Psychology, Franz Hall, University of California, Los Angeles, CA 90095-1563, USA. lieber@ucla.edu
Cunningham
William A
WA
D001921:Brain / Q000098:blood supply*; D001931:Brain Mapping; D003627:Data Interpretation, Statistical; D005189:False Positive Reactions; D006801:Humans; D007090:Image Interpretation, Computer-Assisted; D008279:Magnetic Resonance Imaging; D015201:Meta-Analysis as Topic; D010100:Oxygen / Q000097:blood; D015203:Reproducibility of Results; D012107:Research Design; D018401:Sample Size
D010100:Oxygen