title: Avoiding model selection bias in small-sample genomic datasets.
abstract: BACKGROUND
  Genomic datasets generated by high-throughput technologies are typically characterized by a moderate number of samples and a large number of measurements per sample. As a consequence, classification models are commonly compared based on resampling techniques. This investigation discusses the conceptual difficulties involved in comparative classification studies. Conclusions derived from such studies are often optimistically biased, because the apparent differences in performance are usually not controlled in a statistically stringent framework taking into account the adopted sampling strategy. We investigate this problem by means of a comparison of various classifiers in the context of multiclass microarray data.
  
  
  RESULTS
  Commonly used accuracy-based performance values, with or without confidence intervals, are inadequate for comparing classifiers for small-sample data. We present a statistical methodology that avoids bias in cross-validated model selection in the context of small-sample scenarios. This methodology is valid for both k-fold cross-validation and repeated random sampling.
authors.affiliation: School of Biomedical Sciences, University of Ulster at Coleraine Northern Ireland. dp.berrar@ulster.ac.uk
authors.forename: Daniel
authors.initials: D
authors.lastname: Berrar
authors.forename: Ian
authors.initials: I
authors.lastname: Bradbury
authors.forename: Werner
authors.initials: W
authors.lastname: Dubitzky
mesh_terms: D000465:Algorithms; D015982:Bias; D014408:Biomarkers, Tumor / Q000235:genetics / Q000378:metabolism; D002874:Chromosome Mapping / Q000379:methods; D003198:Computer Simulation; D030541:Databases, Genetic; D003936:Diagnosis, Computer-Assisted / Q000379:methods; D020869:Gene Expression Profiling / Q000379:methods*; D006801:Humans; D008957:Models, Genetic; D009363:Neoplasm Proteins / Q000235:genetics / Q000378:metabolism*; D009369:Neoplasms / Q000175:diagnosis / Q000235:genetics / Q000378:metabolism*; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods*; D015203:Reproducibility of Results; D018401:Sample Size; D012680:Sensitivity and Specificity
chemical_list: D014408:Biomarkers, Tumor; D009363:Neoplasm Proteins