title: Sample size and statistical power considerations in high-dimensionality data settings: a comparative study of classification algorithms.
abstract: BACKGROUND
  data generated using 'omics' technologies are characterized by high dimensionality, where the number of features measured per subject vastly exceeds the number of subjects in the study. In this paper, we consider issues relevant in the design of biomedical studies in which the goal is the discovery of a subset of features and an associated algorithm that can predict a binary outcome, such as disease status. We compare the performance of four commonly used classifiers (K-Nearest Neighbors, Prediction Analysis for Microarrays, Random Forests and Support Vector Machines) in high-dimensionality data settings. We evaluate the effects of varying levels of signal-to-noise ratio in the dataset, imbalance in class distribution and choice of metric for quantifying performance of the classifier. To guide study design, we present a summary of the key characteristics of 'omics' data profiled in several human or animal model experiments utilizing high-content mass spectrometry and multiplexed immunoassay based techniques.
  
  
  RESULTS
  the analysis of data from seven 'omics' studies revealed that the average magnitude of effect size observed in human studies was markedly lower when compared to that in animal studies. The data measured in human studies were characterized by higher biological variation and the presence of outliers. The results from simulation studies indicated that the classifier Prediction Analysis for Microarrays (PAM) had the highest power when the class conditional feature distributions were Gaussian and outcome distributions were balanced. Random Forests was optimal when feature distributions were skewed and when class distributions were unbalanced. We provide a free open-source R statistical software library (MVpower) that implements the simulation strategy proposed in this paper.
  
  
  CONCLUSIONS
  no single classifier had optimal performance under all settings. Simulation studies provide useful guidance for the design of biomedical studies involving high-dimensionality data.
authors.affiliation: BG Medicine, Inc., 610 Lincoln St., Waltham, MA 02451, USA.
authors.forename: Yu
authors.initials: Y
authors.lastname: Guo
authors.forename: Armin
authors.initials: A
authors.lastname: Graber
authors.forename: Robert N
authors.initials: RN
authors.lastname: McBurney
authors.forename: Raji
authors.initials: R
authors.lastname: Balasubramanian
mesh_terms: D000465:Algorithms; D000818:Animals; D002965:Classification / Q000379:methods*; D016208:Databases, Factual; D020869:Gene Expression Profiling / Q000379:methods; D006801:Humans; D015233:Models, Statistical; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods; D010363:Pattern Recognition, Automated; D018401:Sample Size
references.citation: Bioinformatics. 2005 Dec 1;21(23):4263-71
references.pmid: 16204346
references.citation: PLoS One. 2009;4(3):e4922
references.pmid: 19290050
references.citation: Bioinformatics. 2005 Jul 1;21(13):3017-24
references.pmid: 15840707
references.citation: Stat Med. 2002 Dec 15;21(23):3543-70
references.pmid: 12436455
references.citation: Biostatistics. 2005 Jan;6(1):157-69
references.pmid: 15618534
references.citation: Proc Natl Acad Sci U S A. 2002 May 14;99(10):6567-72
references.pmid: 12011421
references.citation: Biomarkers. 2009 Dec;14(8):572-86
references.pmid: 19780643
references.citation: Biostatistics. 2005 Jan;6(1):27-38
references.pmid: 15618525
references.citation: J Comput Biol. 2004;11(4):714-26
references.pmid: 15579240
references.citation: Bioinformatics. 2005 Apr 15;21(8):1502-8
references.pmid: 15564298
references.citation: Biostatistics. 2007 Jan;8(1):101-17
references.pmid: 16613833
references.citation: Toxicol Pathol. 2009 Jan;37(1):52-64
references.pmid: 19171931
references.citation: BMC Genomics. 2004 Nov 08;5:87
references.pmid: 15533245
references.citation: Laryngoscope. 2009 Jul;119(7):1291-302
references.pmid: 19444892
references.citation: Bioinformatics. 2002 Sep;18(9):1184-93
references.pmid: 12217910
references.citation: Proc Natl Acad Sci U S A. 2006 Apr 11;103(15):5923-8
references.pmid: 16585533
references.citation: Stat Med. 2005 Aug 15;24(15):2267-80
references.pmid: 15977294
references.citation: Blood. 2009 Jul 30;114(5):1063-72
references.pmid: 19443663
references.citation: Anal Chem. 2006 Jan 15;78(2):567-74
references.pmid: 16408941