Sample size and statistical power considerations in high-dimensionality data settings: a comparative study of classification algorithms.
BACKGROUND
data generated using 'omics' technologies are characterized by high dimensionality, where the number of features measured per subject vastly exceeds the number of subjects in the study. In this paper, we consider issues relevant in the design of biomedical studies in which the goal is the discovery of a subset of features and an associated algorithm that can predict a binary outcome, such as disease status. We compare the performance of four commonly used classifiers (K-Nearest Neighbors, Prediction Analysis for Microarrays, Random Forests and Support Vector Machines) in high-dimensionality data settings. We evaluate the effects of varying levels of signal-to-noise ratio in the dataset, imbalance in class distribution and choice of metric for quantifying performance of the classifier. To guide study design, we present a summary of the key characteristics of 'omics' data profiled in several human or animal model experiments utilizing high-content mass spectrometry and multiplexed immunoassay based techniques.


RESULTS
the analysis of data from seven 'omics' studies revealed that the average magnitude of effect size observed in human studies was markedly lower when compared to that in animal studies. The data measured in human studies were characterized by higher biological variation and the presence of outliers. The results from simulation studies indicated that the classifier Prediction Analysis for Microarrays (PAM) had the highest power when the class conditional feature distributions were Gaussian and outcome distributions were balanced. Random Forests was optimal when feature distributions were skewed and when class distributions were unbalanced. We provide a free open-source R statistical software library (MVpower) that implements the simulation strategy proposed in this paper.


CONCLUSIONS
no single classifier had optimal performance under all settings. Simulation studies provide useful guidance for the design of biomedical studies involving high-dimensionality data.
Guo
Yu
Y
BG Medicine, Inc., 610 Lincoln St., Waltham, MA 02451, USA.
Graber
Armin
A
McBurney
Robert N
RN
Balasubramanian
Raji
R
D000465:Algorithms; D000818:Animals; D002965:Classification / Q000379:methods*; D016208:Databases, Factual; D020869:Gene Expression Profiling / Q000379:methods; D006801:Humans; D015233:Models, Statistical; D020411:Oligonucleotide Array Sequence Analysis / Q000379:methods; D010363:Pattern Recognition, Automated; D018401:Sample Size
Bioinformatics. 2005 Dec 1;21(23):4263-71
16204346
PLoS One. 2009;4(3):e4922
19290050
Bioinformatics. 2005 Jul 1;21(13):3017-24
15840707
Stat Med. 2002 Dec 15;21(23):3543-70
12436455
Biostatistics. 2005 Jan;6(1):157-69
15618534
Proc Natl Acad Sci U S A. 2002 May 14;99(10):6567-72
12011421
Biomarkers. 2009 Dec;14(8):572-86
19780643
Biostatistics. 2005 Jan;6(1):27-38
15618525
J Comput Biol. 2004;11(4):714-26
15579240
Bioinformatics. 2005 Apr 15;21(8):1502-8
15564298
Biostatistics. 2007 Jan;8(1):101-17
16613833
Toxicol Pathol. 2009 Jan;37(1):52-64
19171931
BMC Genomics. 2004 Nov 08;5:87
15533245
Laryngoscope. 2009 Jul;119(7):1291-302
19444892
Bioinformatics. 2002 Sep;18(9):1184-93
12217910
Proc Natl Acad Sci U S A. 2006 Apr 11;103(15):5923-8
16585533
Stat Med. 2005 Aug 15;24(15):2267-80
15977294
Blood. 2009 Jul 30;114(5):1063-72
19443663
Anal Chem. 2006 Jan 15;78(2):567-74
16408941