title: Over-optimism in bioinformatics: an illustration.
abstract: BACKGROUND
  In statistical bioinformatics research, different optimization mechanisms potentially lead to 'over-optimism' in published papers. So far, however, a systematic critical study concerning the various sources underlying this over-optimism is lacking.
  
  
  RESULTS
  We present an empirical study on over-optimism using high-dimensional classification as example. Specifically, we consider a 'promising' new classification algorithm, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. While this approach yields poor results in terms of error rate, we quantitatively demonstrate that it can artificially seem superior to existing approaches if we 'fish for significance'. The investigated sources of over-optimism include the optimization of datasets, of settings, of competing methods and, most importantly, of the method's characteristics. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should always be demonstrated on independent validation data.
  
  
  BACKGROUND
  The R codes and relevant data can be downloaded from http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/overoptimism/, such that the study is completely reproducible.
authors.affiliation: Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Munich, Germany.
authors.forename: Monika
authors.initials: M
authors.lastname: Jelizarow
authors.forename: Vincent
authors.initials: V
authors.lastname: Guillemot
authors.forename: Arthur
authors.initials: A
authors.lastname: Tenenhaus
authors.forename: Korbinian
authors.initials: K
authors.lastname: Strimmer
authors.forename: Anne-Laure
authors.initials: AL
authors.lastname: Boulesteix
mesh_terms: D000465:Algorithms; D019295:Computational Biology / Q000379:methods*; D016002:Discriminant Analysis; D020869:Gene Expression Profiling / Q000379:methods