title: Effect of training-sample size and classification difficulty on the accuracy of genomic predictors.
abstract: BACKGROUND
  As part of the MicroArray Quality Control (MAQC)-II project, this analysis examines how the choice of univariate feature-selection methods and classification algorithms may influence the performance of genomic predictors under varying degrees of prediction difficulty represented by three clinically relevant endpoints.
  
  
  METHODS
  We used gene-expression data from 230 breast cancers (grouped into training and independent validation sets), and we examined 40 predictors (five univariate feature-selection methods combined with eight different classifiers) for each of the three endpoints. Their classification performance was estimated on the training set by using two different resampling methods and compared with the accuracy observed in the independent validation set.
  
  
  RESULTS
  A ranking of the three classification problems was obtained, and the performance of 120 models was estimated and assessed on an independent validation set. The bootstrapping estimates were closer to the validation performance than were the cross-validation estimates. The required sample size for each endpoint was estimated, and both gene-level and pathway-level analyses were performed on the obtained models.
  
  
  CONCLUSIONS
  We showed that genomic predictor accuracy is determined largely by an interplay between sample size and classification difficulty. Variations on univariate feature-selection methods and choice of classification algorithm have only a modest impact on predictor performance, and several statistically equally good predictors can be developed for any given classification problem.
authors.affiliation: Bioinformatics Core Facility, Swiss Institute of Bioinformatics, Génopode Building, Quartier Sorge, Lausanne CH-1015, Switzerland.
authors.forename: Vlad
authors.initials: V
authors.lastname: Popovici
authors.forename: Weijie
authors.initials: W
authors.lastname: Chen
authors.forename: Brandon G
authors.initials: BG
authors.lastname: Gallas
authors.forename: Christos
authors.initials: C
authors.lastname: Hatzis
authors.forename: Weiwei
authors.initials: W
authors.lastname: Shi
authors.forename: Frank W
authors.initials: FW
authors.lastname: Samuelson
authors.forename: Yuri
authors.initials: Y
authors.lastname: Nikolsky
authors.forename: Marina
authors.initials: M
authors.lastname: Tsyganova
authors.forename: Alex
authors.initials: A
authors.lastname: Ishkin
authors.forename: Tatiana
authors.initials: T
authors.lastname: Nikolskaya
authors.forename: Kenneth R
authors.initials: KR
authors.lastname: Hess
authors.forename: Vicente
authors.initials: V
authors.lastname: Valero
authors.forename: Daniel
authors.initials: D
authors.lastname: Booser
authors.forename: Mauro
authors.initials: M
authors.lastname: Delorenzi
authors.forename: Gabriel N
authors.initials: GN
authors.lastname: Hortobagyi
authors.forename: Leming
authors.initials: L
authors.lastname: Shi
authors.forename: W Fraser
authors.initials: WF
authors.lastname: Symmans
authors.forename: Lajos
authors.initials: L
authors.lastname: Pusztai
mesh_terms: D000465:Algorithms; D019540:Area Under Curve; D001943:Breast Neoplasms / Q000737:chemistry / Q000235:genetics*; D005260:Female; D020869:Gene Expression Profiling / Q000379:methods*; D006801:Humans; D011960:Receptors, Estrogen / Q000032:analysis; D018401:Sample Size
chemical_list: D011960:Receptors, Estrogen