Over-optimism in bioinformatics: an illustration.
BACKGROUND
In statistical bioinformatics research, different optimization mechanisms potentially lead to 'over-optimism' in published papers. So far, however, a systematic critical study concerning the various sources underlying this over-optimism is lacking.


RESULTS
We present an empirical study on over-optimism using high-dimensional classification as example. Specifically, we consider a 'promising' new classification algorithm, namely linear discriminant analysis incorporating prior knowledge on gene functional groups through an appropriate shrinkage of the within-group covariance matrix. While this approach yields poor results in terms of error rate, we quantitatively demonstrate that it can artificially seem superior to existing approaches if we 'fish for significance'. The investigated sources of over-optimism include the optimization of datasets, of settings, of competing methods and, most importantly, of the method's characteristics. We conclude that, if the improvement of a quantitative criterion such as the error rate is the main contribution of a paper, the superiority of new algorithms should always be demonstrated on independent validation data.


BACKGROUND
The R codes and relevant data can be downloaded from http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/overoptimism/, such that the study is completely reproducible.
Jelizarow
Monika
M
Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Munich, Germany.
Guillemot
Vincent
V
Tenenhaus
Arthur
A
Strimmer
Korbinian
K
Boulesteix
Anne-Laure
AL
D000465:Algorithms; D019295:Computational Biology / Q000379:methods*; D016002:Discriminant Analysis; D020869:Gene Expression Profiling / Q000379:methods