title: Cross-study validation for the assessment of prediction algorithms.
abstract: BACKGROUND
  Numerous competing algorithms for prediction in high-dimensional settings have been developed in the statistical and machine-learning literature. Learning algorithms and the prediction models they generate are typically evaluated on the basis of cross-validation error estimates in a few exemplary datasets. However, in most applications, the ultimate goal of prediction modeling is to provide accurate predictions for independent samples obtained in different settings. Cross-validation within exemplary datasets may not adequately reflect performance in the broader application context.
  
  
  METHODS
  We develop and implement a systematic approach to 'cross-study validation', to replace or supplement conventional cross-validation when evaluating high-dimensional prediction models in independent datasets. We illustrate it via simulations and in a collection of eight estrogen-receptor positive breast cancer microarray gene-expression datasets, where the objective is predicting distant metastasis-free survival (DMFS). We computed the C-index for all pairwise combinations of training and validation datasets. We evaluate several alternatives for summarizing the pairwise validation statistics, and compare these to conventional cross-validation.
  
  
  RESULTS
  Our data-driven simulations and our application to survival prediction with eight breast cancer microarray datasets, suggest that standard cross-validation produces inflated discrimination accuracy for all algorithms considered, when compared to cross-study validation. Furthermore, the ranking of learning algorithms differs, suggesting that algorithms performing best in cross-validation may be suboptimal when evaluated through independent validation.
  
  
  BACKGROUND
  The survHD: Survival in High Dimensions package (http://www.bitbucket.org/lwaldron/survhd) will be made available through Bioconductor.
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USALeibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Christoph
authors.initials: C
authors.lastname: Bernau
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USALeibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Markus
authors.initials: M
authors.lastname: Riester
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Anne-Laure
authors.initials: AL
authors.lastname: Boulesteix
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USALeibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Giovanni
authors.initials: G
authors.lastname: Parmigiani
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Curtis
authors.initials: C
authors.lastname: Huttenhower
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Levi
authors.initials: L
authors.lastname: Waldron
authors.affiliation: Leibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USALeibniz Supercomputing Center, Garching, Department for Medical Informatics, Biometry and Epidemiology, Munich, Germany, Cambridge, MA, Dana-Farber Cancer Institute, Boston, Harvard School of Public Health, Boston, USA and City University of New York School of Public Health, Hunter College, New York, USA.
authors.forename: Lorenzo
authors.initials: L
authors.lastname: Trippa
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D001943:Breast Neoplasms / Q000473:pathology; D018572:Disease-Free Survival; D005260:Female; D020869:Gene Expression Profiling; D006801:Humans; D009362:Neoplasm Metastasis; D020411:Oligonucleotide Array Sequence Analysis; D011379:Prognosis; D015203:Reproducibility of Results