title: A plea for neutral comparison studies in computational sciences.
abstract: In computational science literature including, e.g., bioinformatics, computational statistics or machine learning, most published articles are devoted to the development of "new methods", while comparison studies are generally appreciated by readers but surprisingly given poor consideration by many journals. This paper stresses the importance of neutral comparison studies for the objective evaluation of existing methods and the establishment of standards by drawing parallels with clinical research. The goal of the paper is twofold. Firstly, we present a survey of recent computational papers on supervised classification published in seven high-ranking computational science journals. The aim is to provide an up-to-date picture of current scientific practice with respect to the comparison of methods in both articles presenting new methods and articles focusing on the comparison study itself. Secondly, based on the results of our survey we critically discuss the necessity, impact and limitations of neutral comparison studies in computational sciences. We define three reasonable criteria a comparison study has to fulfill in order to be considered as neutral, and explicate general considerations on the individual components of a "tidy neutral comparison study". R codes for completely replicating our statistical analyses and figures are available from the companion website http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/020_professuren/boulesteix/plea2013.
authors: Department of Medical Informatics, Biometry and Epidemiology, Ludwig-Maximilians-University of Munich, Munich, Germany. boulesteix@ibe.med.uni-muenchen.de
authors: Anne-Laure
authors: AL
authors: Boulesteix
authors: Sabine
authors: S
authors: Lauer
authors: Manuel J A
authors: MJ
authors: Eugster
mesh_terms: D001185:Artificial Intelligence; D019295:Computational Biology / Q000379:methods; D006801:Humans
references: Bioinformatics. 2010 Feb 1;26(3):437-9
references: 19942585
references: Brief Bioinform. 2011 May;12(3):288-300
references: 21278369
references: Biostatistics. 2010 Jul;11(3):376-8
references: 20498225
references: Bioinformatics. 2010 Aug 15;26(16):1990-8
references: 20581402
references: Bioinformatics. 2010 Jan 1;26(1):68-76
references: 19846436
references: PLoS Med. 2009 Sep;6(9):e1000144
references: 19901971