title: A measure of the impact of CV incompleteness on prediction error estimation with application to PCA and normalization.
abstract: BACKGROUND
  In applications of supervised statistical learning in the biomedical field it is necessary to assess the prediction error of the respective prediction rules. Often, data preparation steps are performed on the dataset-in its entirety-before training/test set based prediction error estimation by cross-validation (CV)-an approach referred to as "incomplete CV". Whether incomplete CV can result in an optimistically biased error estimate depends on the data preparation step under consideration. Several empirical studies have investigated the extent of bias induced by performing preliminary supervised variable selection before CV. To our knowledge, however, the potential bias induced by other data preparation steps has not yet been examined in the literature. In this paper we investigate this bias for two common data preparation steps: normalization and principal component analysis for dimension reduction of the covariate space (PCA). Furthermore we obtain preliminary results for the following steps: optimization of tuning parameters, variable filtering by variance and imputation of missing values.
  
  
  METHODS
  We devise the easily interpretable and general measure CVIIM ("CV Incompleteness Impact Measure") to quantify the extent of bias induced by incomplete CV with respect to a data preparation step of interest. This measure can be used to determine whether a specific data preparation step should, as a general rule, be performed in each CV iteration or whether an incomplete CV procedure would be acceptable in practice. We apply CVIIM to large collections of microarray datasets to answer this question for normalization and PCA.
  
  
  RESULTS
  Performing normalization on the entire dataset before CV did not result in a noteworthy optimistic bias in any of the investigated cases. In contrast, when performing PCA before CV, medium to strong underestimates of the prediction error were observed in multiple settings.
  
  
  CONCLUSIONS
  While the investigated forms of normalization can be safely performed before CV, PCA has to be performed anew in each CV split to protect against optimistic bias.
authors.affiliation: Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, Munich, D-81377, Germany. hornung@ibe.med.uni-muenchen.de.
authors.forename: Roman
authors.initials: R
authors.lastname: Hornung
authors.affiliation: Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, Munich, D-81377, Germany. Christoph.Bernau@lrz.de.|Leibniz Supercomputing Center, Boltzmannstr. 1, Garching, D-85748, Germany. Christoph.Bernau@lrz.de.
authors.forename: Christoph
authors.initials: C
authors.lastname: Bernau
authors.affiliation: Clinical and Innovation Proteomic Platform, Pôle de Recherche Université de Bourgogne, 15 Bd Maréchal de Lattre de Tassigny, Dijon, F-21000, France. Caroline.TRUNTZER@clipproteomic.fr.
authors.forename: Caroline
authors.initials: C
authors.lastname: Truntzer
authors.affiliation: Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, Munich, D-81377, Germany. wilson.rory@gmail.com.
authors.forename: Rory
authors.initials: R
authors.lastname: Wilson
authors.affiliation: Department of Urology, University of Munich, Marchioninistr. 15, Munich, D-81377, Germany. praxis@urologie-im-tal.de.
authors.forename: Thomas
authors.initials: T
authors.lastname: Stadler
authors.affiliation: Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, Munich, D-81377, Germany. boulesteix@ibe.med.uni-muenchen.de.
authors.forename: Anne-Laure
authors.initials: AL
authors.lastname: Boulesteix
mesh_terms: D000465:Algorithms; D003627:Data Interpretation, Statistical; D006801:Humans; D020411:Oligonucleotide Array Sequence Analysis; D025341:Principal Component Analysis; D012044:Regression Analysis; D015983:Selection Bias