title: The quantitative evaluation of functional neuroimaging experiments: the NPAIRS data analysis framework.
abstract: We introduce a data-analysis framework and performance metrics for evaluating and optimizing the interaction between activation tasks, experimental designs, and the methodological choices and tools for data acquisition, preprocessing, data analysis, and extraction of statistical parametric maps (SPMs). Our NPAIRS (nonparametric prediction, activation, influence, and reproducibility resampling) framework provides an alternative to simulations and ROC curves by using real PET and fMRI data sets to examine the relationship between prediction accuracy and the signal-to-noise ratios (SNRs) associated with reproducible SPMs. Using cross-validation resampling we plot training-test set predictions of the experimental design variables (e.g., brain-state labels) versus reproducibility SNR metrics for the associated SPMs. We demonstrate the utility of this framework across the wide range of performance metrics obtained from [(15)O]water PET studies of 12 age- and sex-matched data sets performing different motor tasks (8 subjects/set). For the 12 data sets we apply NPAIRS with both univariate and multivariate data-analysis approaches to: (1) demonstrate that this framework may be used to obtain reproducible SPMs from any data-analysis approach on a common Z-score scale (rSPM[Z]); (2) demonstrate that the histogram of a rSPM[Z] image may be modeled as the sum of a data-analysis-dependent noise distribution and a task-dependent, Gaussian signal distribution that scales monotonically with our reproducibility performance metric; (3) explore the relation between prediction and reproducibility performance metrics with an emphasis on bias-variance tradeoffs for flexible, multivariate models; and (4) measure the broad range of reproducibility SNRs and the significant influence of individual subjects. A companion paper describes learning curves for four of these 12 data sets, which describe an alternative mutual-information prediction metric and NPAIRS reproducibility as a function of training-set sizes from 2 to 18 subjects. We propose the NPAIRS framework as a validation tool for testing and optimizing methodological choices and tools in functional neuroimaging.
authors.affiliation: Department of Radiology, University of Minnesota, Minneapolis, Minnesota 55455, USA.
authors.forename: Stephen C
authors.initials: SC
authors.lastname: Strother
authors.forename: Jon
authors.initials: J
authors.lastname: Anderson
authors.forename: Lars Kai
authors.initials: LK
authors.lastname: Hansen
authors.forename: Ulrik
authors.initials: U
authors.lastname: Kjems
authors.forename: Rafal
authors.initials: R
authors.lastname: Kustra
authors.forename: John
authors.initials: J
authors.lastname: Sidtis
authors.forename: Sally
authors.initials: S
authors.lastname: Frutiger
authors.forename: Suraj
authors.initials: S
authors.lastname: Muley
authors.forename: Stephen
authors.initials: S
authors.lastname: LaConte
authors.forename: David
authors.initials: D
authors.lastname: Rottenberg
mesh_terms: D000328:Adult; D001288:Attention / Q000502:physiology; D001931:Brain Mapping / Q000379:methods*; D002540:Cerebral Cortex / Q000502:physiology*; D003627:Data Interpretation, Statistical; D004292:Dominance, Cerebral / Q000502:physiology; D005260:Female; D006801:Humans; D007091:Image Processing, Computer-Assisted; D021621:Imaging, Three-Dimensional; D008279:Magnetic Resonance Imaging / Q000706:statistics & numerical data*; D008297:Male; D008432:Mathematical Computing; D008875:Middle Aged; D015233:Models, Statistical; D010700:Phonetics; D011597:Psychomotor Performance / Q000502:physiology*; D012016:Reference Values; D015203:Reproducibility of Results; D013067:Speech Perception / Q000502:physiology*; D014055:Tomography, Emission-Computed / Q000706:statistics & numerical data*