title: Batch effect confounding leads to strong bias in performance estimates obtained by cross-validation.
abstract: BACKGROUND
  With the large amount of biological data that is currently publicly available, many investigators combine multiple data sets to increase the sample size and potentially also the power of their analyses. However, technical differences ("batch effects") as well as differences in sample composition between the data sets may significantly affect the ability to draw generalizable conclusions from such studies.
  
  
  OBJECTIVE
  The current study focuses on the construction of classifiers, and the use of cross-validation to estimate their performance. In particular, we investigate the impact of batch effects and differences in sample composition between batches on the accuracy of the classification performance estimate obtained via cross-validation. The focus on estimation bias is a main difference compared to previous studies, which have mostly focused on the predictive performance and how it relates to the presence of batch effects.
  
  
  METHODS
  We work on simulated data sets. To have realistic intensity distributions, we use real gene expression data as the basis for our simulation. Random samples from this expression matrix are selected and assigned to group 1 (e.g., 'control') or group 2 (e.g., 'treated'). We introduce batch effects and select some features to be differentially expressed between the two groups. We consider several scenarios for our study, most importantly different levels of confounding between groups and batch effects.
  
  
  METHODS
  We focus on well-known classifiers: logistic regression, Support Vector Machines (SVM), k-nearest neighbors (kNN) and Random Forests (RF). Feature selection is performed with the Wilcoxon test or the lasso. Parameter tuning and feature selection, as well as the estimation of the prediction performance of each classifier, is performed within a nested cross-validation scheme. The estimated classification performance is then compared to what is obtained when applying the classifier to independent data.
authors: SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland.
authors: Charlotte
authors: C
authors: Soneson
authors: SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland.
authors: Sarah
authors: S
authors: Gerster
authors: SIB Swiss Institute of Bioinformatics, Lausanne, Switzerland; Ludwig Center for Cancer Research, University of Lausanne, Lausanne, Switzerland; Oncology Department, University of Lausanne, Lausanne, Switzerland.
authors: Mauro
authors: M
authors: Delorenzi
mesh_terms: D001185:Artificial Intelligence; D015982:Bias; D019295:Computational Biology; D020869:Gene Expression Profiling; D015203:Reproducibility of Results; D013223:Statistics as Topic / Q000379:methods*