title: Strengths and limitations of microarray-based phenotype prediction: lessons learned from the IMPROVER Diagnostic Signature Challenge.
abstract: BACKGROUND
  After more than a decade since microarrays were used to predict phenotype of biological samples, real-life applications for disease screening and identification of patients who would best benefit from treatment are still emerging. The interest of the scientific community in identifying best approaches to develop such prediction models was reaffirmed in a competition style international collaboration called IMPROVER Diagnostic Signature Challenge whose results we describe herein.
  
  
  RESULTS
  Fifty-four teams used public data to develop prediction models in four disease areas including multiple sclerosis, lung cancer, psoriasis and chronic obstructive pulmonary disease, and made predictions on blinded new data that we generated. Teams were scored using three metrics that captured various aspects of the quality of predictions, and best performers were awarded. This article presents the challenge results and introduces to the community the approaches of the best overall three performers, as well as an R package that implements the approach of the best overall team. The analyses of model performance data submitted in the challenge as well as additional simulations that we have performed revealed that (i) the quality of predictions depends more on the disease endpoint than on the particular approaches used in the challenge; (ii) the most important modeling factor (e.g. data preprocessing, feature selection and classifier type) is problem dependent; and (iii) for optimal results datasets and methods have to be carefully matched. Biomedical factors such as the disease severity and confidence in diagnostic were found to be associated with the misclassification rates across the different teams.
  
  
  BACKGROUND
  The lung cancer dataset is available from Gene Expression Omnibus (accession, GSE43580). The maPredictDSC R package implementing the approach of the best overall team is available at www.bioconductor.org or http://bioinformaticsprb.med.wayne.edu/.
authors: Department of Computer Science, Wayne State University, Perinatology Research Branch, NICHD/NIH, Detroit, MI 48201, USA, The Microsoft Research - University of Trento Centre for Computational and Systems Biology, Rovereto 38068, Italy, ETH Zurich, Zurich 8092, Switzerland, IBM Thomas J. Watson Research Center, Yorktown Heights, NY 10598, USA and Philip Morris International, Research & Development, Neuchâtel CH-2000, Switzerland.
authors: Adi L
authors: AL
authors: Tarca
authors: Mario
authors: M
authors: Lauria
authors: Michael
authors: M
authors: Unger
authors: Erhan
authors: E
authors: Bilal
authors: Stephanie
authors: S
authors: Boue
authors: Kushal
authors: K
authors: Kumar Dey
authors: Julia
authors: J
authors: Hoeng
authors: Heinz
authors: H
authors: Koeppl
authors: Florian
authors: F
authors: Martin
authors: Pablo
authors: P
authors: Meyer
authors: Preetam
authors: P
authors: Nandy
authors: Raquel
authors: R
authors: Norel
authors: Manuel
authors: M
authors: Peitsch
authors: Jeremy J
authors: JJ
authors: Rice
authors: Roberto
authors: R
authors: Romero
authors: Gustavo
authors: G
authors: Stolovitzky
authors: Marja
authors: M
authors: Talikka
authors: Yang
authors: Y
authors: Xiang
authors: Christoph
authors: C
authors: Zechner
mesh_terms: D004194:Disease / Q000235:genetics; D020869:Gene Expression Profiling / Q000379:methods*; D006801:Humans; D008175:Lung Neoplasms / Q000175:diagnosis / Q000235:genetics; D025202:Molecular Diagnostic Techniques; D009103:Multiple Sclerosis / Q000175:diagnosis / Q000235:genetics; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods*; D010641:Phenotype; D011565:Psoriasis / Q000175:diagnosis / Q000235:genetics; D029424:Pulmonary Disease, Chronic Obstructive / Q000175:diagnosis / Q000235:genetics
references: Clin Transplant. 2007 Jul-Aug;21(4):523-5
references: 17645713
references: Biochim Biophys Acta. 1975 Oct 20;405(2):442-51
references: 1180967
references: J Am Acad Dermatol. 2004 Oct;51(4):563-9
references: 15389191
references: Nature. 2000 Aug 17;406(6797):747-52
references: 10963602
references: Nature. 2000 Aug 3;406(6795):536-40
references: 10952317
references: Science. 1995 Oct 20;270(5235):467-70
references: 7569999
references: Nat Biotechnol. 2010 Aug;28(8):827-38
references: 20676074
references: Pharmacogenomics J. 2010 Aug;10(4):278-91
references: 20676067
references: PLoS Comput Biol. 2007 Jun;3(6):e116
references: 17604446
references: Biostatistics. 2003 Apr;4(2):249-64
references: 12925520
references: J Pers Med. 2012 Oct 30;2(4):201-16
references: 25562360
references: J Heart Lung Transplant. 2007 Apr;26(4):403-6
references: 17403484
references: Nat Biotechnol. 2011 Sep 08;29(9):811-5
references: 21904331
references: Genome Res. 2003 Nov;13(11):2498-504
references: 14597658
references: Expert Rev Mol Diagn. 2006 Nov;6(6):803-9
references: 17140367
references: Ann Surg Oncol. 2012 Oct;19(10):3257-63
references: 22965266
references: Cancer Genomics Proteomics. 2007 May-Jun;4(3):147-55
references: 17878518
references: Nat Methods. 2012 Jul 15;9(8):796-804
references: 22796662
references: Bioinformatics. 2012 May 1;28(9):1193-201
references: 22423044
references: Bioinformatics. 2000 May;16(5):412-24
references: 10871264
references: Cancer Res. 2002 Sep 1;62(17):4963-7
references: 12208747
references: PLoS One. 2010 Feb 23;5(2):e9202
references: 20186320
references: Oncology (Williston Park). 2006 Jun;20(7):789-90
references: 16841801