title: An AUC-based permutation variable importance measure for random forests.
abstract: BACKGROUND
  The random forest (RF) method is a commonly used tool for classification with high dimensional data as well as for ranking candidate predictors based on the so-called random forest variable importance measures (VIMs). However the classification performance of RF is known to be suboptimal in case of strongly unbalanced data, i.e. data where response class sizes differ considerably. Suggestions were made to obtain better classification performance based either on sampling procedures or on cost sensitivity analyses. However to our knowledge the performance of the VIMs has not yet been examined in the case of unbalanced response classes. In this paper we explore the performance of the permutation VIM for unbalanced data settings and introduce an alternative permutation VIM based on the area under the curve (AUC) that is expected to be more robust towards class imbalance.
  
  
  RESULTS
  We investigated the performance of the standard permutation VIM and of our novel AUC-based permutation VIM for different class imbalance levels using simulated data and real data. The results suggest that the new AUC-based permutation VIM outperforms the standard permutation VIM for unbalanced data settings while both permutation VIMs have equal performance for balanced data settings.
  
  
  CONCLUSIONS
  The standard permutation VIM loses its ability to discriminate between associated predictors and predictors not associated with the response for increasing class imbalance. It is outperformed by our new AUC-based permutation VIM for unbalanced data settings, while the performance of both VIMs is very similar in the case of balanced classes. The new AUC-based VIM is implemented in the R package party for the unbiased RF variant based on conditional inference trees. The codes implementing our study are available from the companion website: http://www.ibe.med.uni-muenchen.de/organisation/mitarbeiter/070_drittmittel/janitza/index.html.
authors.affiliation: Department of Medical Informatics, Biometry and Epidemiology, University of Munich, Marchioninistr. 15, D-81377, Munich, Germany. janitza@ibe.med.uni-muenchen.de
authors.forename: Silke
authors.initials: S
authors.lastname: Janitza
authors.forename: Carolin
authors.initials: C
authors.lastname: Strobl
authors.forename: Anne-Laure
authors.initials: AL
authors.lastname: Boulesteix
mesh_terms: D000465:Algorithms; D019540:Area Under Curve; D017393:RNA Editing; D018401:Sample Size
references.citation: Cancer Epidemiol Biomarkers Prev. 2008 Jun;17(6):1368-73
references.pmid: 18559551
references.citation: BMC Bioinformatics. 2004 Sep 16;5:132
references.pmid: 15373947
references.citation: BMC Bioinformatics. 2010 Feb 27;11:110
references.pmid: 20187966
references.citation: Brief Bioinform. 2013 Jan;14(1):13-26
references.pmid: 22408190
references.citation: Hum Hered. 2011;72(2):121-32
references.pmid: 21996641
references.citation: Brief Bioinform. 2012 May;13(3):292-304
references.pmid: 21908865
references.citation: Bioinformatics. 2009 Aug 1;25(15):1884-90
references.pmid: 19460890
references.citation: BMC Bioinformatics. 2010 Oct 20;11:523
references.pmid: 20961420
references.citation: BMC Proc. 2007;1 Suppl 1:S62
references.pmid: 18466563
references.citation: BMC Med Inform Decis Mak. 2011 Jul 29;11:51
references.pmid: 21801360
references.citation: Hum Genet. 2010 Apr;127(4):441-52
references.pmid: 20084519
references.citation: Hum Genet. 2011 May;129(5):473-85
references.pmid: 21210282
references.citation: BMC Bioinformatics. 2007 Jan 25;8:25
references.pmid: 17254353
references.citation: Am J Epidemiol. 2010 Jul 15;172(2):217-24
references.pmid: 20522537
references.citation: Brief Bioinform. 2011 Jul;12(4):369-73
references.pmid: 21498552