title: A comparison of machine learning methods for classification using simulation with multiple real data examples from mental health studies.
abstract: Recent literature on the comparison of machine learning methods has raised questions about the neutrality, unbiasedness and utility of many comparative studies. Reporting of results on favourable datasets and sampling error in the estimated performance measures based on single samples are thought to be the major sources of bias in such comparisons. Better performance in one or a few instances does not necessarily imply so on an average or on a population level and simulation studies may be a better alternative for objectively comparing the performances of machine learning algorithms.
  
  
  
  We compare the classification performance of a number of important and widely used machine learning algorithms, namely the Random Forests (RF), Support Vector Machines (SVM), Linear Discriminant Analysis (LDA) and k-Nearest Neighbour (kNN). Using massively parallel processing on high-performance supercomputers, we compare the generalisation errors at various combinations of levels of several factors: number of features, training sample size, biological variation, experimental variation, effect size, replication and correlation between features.
  
  
  
  For smaller number of correlated features, number of features not exceeding approximately half the sample size, LDA was found to be the method of choice in terms of average generalisation errors as well as stability (precision) of error estimates. SVM (with RBF kernel) outperforms LDA as well as RF and kNN by a clear margin as the feature set gets larger provided the sample size is not too small (at least 20). The performance of kNN also improves as the number of features grows and outplays that of LDA and RF unless the data variability is too high and/or effect sizes are too small. RF was found to outperform only kNN in some instances where the data are more variable and have smaller effect sizes, in which cases it also provide more stable error estimates than kNN and LDA. Applications to a number of real datasets supported the findings from the simulation study.
authors.affiliation: King's College London, Institute of Psychiatry, Department of Biostatistics, London, UK King's College London, Institute of Psychiatry, NIHR Biomedical Research Centre for Mental Health at the South London and Maudsley NHS Foundation Trust, London, UK mizanur.khondoker@kcl.ac.uk.
authors.forename: Mizanur
authors.initials: M
authors.lastname: Khondoker
authors.affiliation: King's College London, Institute of Psychiatry, NIHR Biomedical Research Centre for Mental Health at the South London and Maudsley NHS Foundation Trust, London, UK King's College London, Institute of Psychiatry, NIHR Biomedical Research Unit for Dementia at the South London and Maudsley NHS Foundation Trust, London, UK.
authors.forename: Richard
authors.initials: R
authors.lastname: Dobson
authors.affiliation: King's College London, Institute of Psychiatry, MRC Social, Genetic and Developmental Psychiatry Centre, UK.
authors.forename: Caroline
authors.initials: C
authors.lastname: Skirrow
authors.affiliation: King's College London, Institute of Psychiatry, NIHR Biomedical Research Centre for Mental Health at the South London and Maudsley NHS Foundation Trust, London, UK King's College London, Institute of Psychiatry, NIHR Biomedical Research Unit for Dementia at the South London and Maudsley NHS Foundation Trust, London, UK.
authors.forename: Andrew
authors.initials: A
authors.lastname: Simmons
authors.affiliation: King's College London, Institute of Psychiatry, Department of Biostatistics, London, UK.
authors.forename: Daniel
authors.initials: D
authors.lastname: Stahl
mesh_terms: D035841:Behavioral Research / Q000379:methods*; D015179:Colorectal Neoplasms; D016002:Discriminant Analysis; D006207:Half-Life; D006801:Humans; D000069550:Machine Learning; D008603:Mental Health; D018401:Sample Size; D060388:Support Vector Machine
keywords: cross-validation; electroencephalogram (EEG); generalisation error; machine learning; magnetic resonance imaging (MRI); microarrays; truncated distribution
references.citation: J Biopharm Stat. 2004 Nov;14(4):1065-84
references.pmid: 15587980
references.citation: J Bioinform Comput Biol. 2010 Dec;8(6):945-65
references.pmid: 21121020
references.citation: Mol Psychiatry. 2006 Oct;11(10):965-78
references.pmid: 16894394
references.citation: J Affect Disord. 2013 May;147(1-3):80-6
references.pmid: 23218897
references.citation: Behav Brain Funct. 2010 Oct 28;6:66
references.pmid: 21029446
references.citation: BMC Bioinformatics. 2008 Jul 22;9:319
references.pmid: 18647401
references.citation: Bioinformatics. 2005 Mar 1;21(5):631-43
references.pmid: 15374862
references.citation: Cancer Inform. 2008;6:77-97
references.pmid: 19259405
references.citation: Analyst. 2007 Dec;132(12):1200-9
references.pmid: 18318280
references.citation: Multivariate Behav Res. 2000 Apr 1;35(2):137-67
references.pmid: 26754081
references.citation: Bioinformatics. 2010 Jan 1;26(1):68-76
references.pmid: 19846436
references.citation: Brain Topogr. 2013 Jan;26(1):9-23
references.pmid: 22890700
references.citation: Bioinformatics. 2005 Apr 15;21(8):1509-15
references.pmid: 15572470
references.citation: J Child Psychol Psychiatry. 2012 Jun;53(6):706-15
references.pmid: 22118296
references.citation: BMC Genomics. 2008;9 Suppl 1:S13
references.pmid: 18366602
references.citation: BMC Bioinformatics. 2005 Aug 24;6:205
references.pmid: 16120216
references.citation: Int J Geriatr Psychiatry. 2011 Jan;26(1):75-82
references.pmid: 21157852
references.citation: BMC Bioinformatics. 2006 Jan 06;7:3
references.pmid: 16398926
references.citation: Bioinformatics. 2011 Jun 15;27(12):1675-83
references.pmid: 21546390
references.citation: Cancer Inform. 2007 Jul 19;3:295-305
references.pmid: 19455249
references.citation: Ann N Y Acad Sci. 2009 Oct;1180:47-55
references.pmid: 19906260
references.citation: Stat Appl Genet Mol Biol. 2004;3:Article3
references.pmid: 16646809