title: On the generalizability of resting-state fMRI machine learning classifiers.
abstract: Machine learning classifiers have become increasingly popular tools to generate single-subject inferences from fMRI data. With this transition from the traditional group level difference investigations to single-subject inference, the application of machine learning methods can be seen as a considerable step forward. Existing studies, however, have given scarce or no information on the generalizability to other subject samples, limiting the use of such published classifiers in other research projects. We conducted a simulation study using publicly available resting-state fMRI data from the 1000 Functional Connectomes and COBRE projects to examine the generalizability of classifiers based on regional homogeneity of resting-state time series. While classification accuracies of up to 0.8 (using sex as the target variable) could be achieved on test datasets drawn from the same study as the training dataset, the generalizability of classifiers to different study samples proved to be limited albeit above chance. This shows that on the one hand a certain amount of generalizability can robustly be expected, but on the other hand this generalizability should not be overestimated. Indeed, this study substantiates the need to include data from several sites in a study investigating machine learning classifiers with the aim of generalizability.
authors.affiliation: Center for Medical Physics and Biomedical Engineering, Medical University of Vienna Vienna, Austria ; MR Centre of Excellence, Medical University of Vienna Vienna, Austria ; Department of Statistics and Probability Theory, Vienna University of Technology Vienna, Austria.
authors.forename: Wolfgang
authors.initials: W
authors.lastname: Huf
authors.affiliation: Center for Medical Physics and Biomedical Engineering, Medical University of Vienna Vienna, Austria ; MR Centre of Excellence, Medical University of Vienna Vienna, Austria ; Department of Statistics and Probability Theory, Vienna University of Technology Vienna, Austria.
authors.forename: Klaudius
authors.initials: K
authors.lastname: Kalcher
authors.affiliation: Center for Medical Physics and Biomedical Engineering, Medical University of Vienna Vienna, Austria ; MR Centre of Excellence, Medical University of Vienna Vienna, Austria ; Department of Statistics and Probability Theory, Vienna University of Technology Vienna, Austria.
authors.forename: Roland N
authors.initials: RN
authors.lastname: Boubela
authors.affiliation: MR Centre of Excellence, Medical University of Vienna Vienna, Austria ; Department of Radiodiagnostics and Nuclear Medicine, Medical University of Vienna Vienna, Austria.
authors.forename: Georg
authors.initials: G
authors.lastname: Rath
authors.affiliation: Department of Pediatrics and Adolescent Medicine, St. Anna Children's Hospital, Medical University of Vienna Vienna, Austria.
authors.forename: Andreas
authors.initials: A
authors.lastname: Vecsei
authors.affiliation: Department of Statistics and Probability Theory, Vienna University of Technology Vienna, Austria.
authors.forename: Peter
authors.initials: P
authors.lastname: Filzmoser
authors.affiliation: Center for Medical Physics and Biomedical Engineering, Medical University of Vienna Vienna, Austria ; MR Centre of Excellence, Medical University of Vienna Vienna, Austria ; Department of Psychiatry, University of Pennsylvania Medical Center Philadelphia, PA, USA.
authors.forename: Ewald
authors.initials: E
authors.lastname: Moser
keywords: classification; machine learning; regional homogeneity; resting-state fMRI; support vector machines