title: An experimental study of the intrinsic stability of random forest variable importance measures.
abstract: BACKGROUND
  The stability of Variable Importance Measures (VIMs) based on random forest has recently received increased attention. Despite the extensive attention on traditional stability of data perturbations or parameter variations, few studies include influences coming from the intrinsic randomness in generating VIMs, i.e. bagging, randomization and permutation. To address these influences, in this paper we introduce a new concept of intrinsic stability of VIMs, which is defined as the self-consistence among feature rankings in repeated runs of VIMs without data perturbations and parameter variations. Two widely used VIMs, i.e., Mean Decrease Accuracy (MDA) and Mean Decrease Gini (MDG) are comprehensively investigated. The motivation of this study is two-fold. First, we empirically verify the prevalence of intrinsic stability of VIMs over many real-world datasets to highlight that the instability of VIMs does not originate exclusively from data perturbations or parameter variations, but also stems from the intrinsic randomness of VIMs. Second, through Spearman and Pearson tests we comprehensively investigate how different factors influence the intrinsic stability.
  
  
  RESULTS
  The experiments are carried out on 19 benchmark datasets with diverse characteristics, including 10 high-dimensional and small-sample gene expression datasets. Experimental results demonstrate the prevalence of intrinsic stability of VIMs. Spearman and Pearson tests on the correlations between intrinsic stability and different factors show that #feature (number of features) and #sample (size of sample) have a coupling effect on the intrinsic stability. The synthetic indictor, #feature/#sample, shows both negative monotonic correlation and negative linear correlation with the intrinsic stability, while OOB accuracy has monotonic correlations with intrinsic stability. This indicates that high-dimensional, small-sample and high complexity datasets may suffer more from intrinsic instability of VIMs. Furthermore, with respect to parameter settings of random forest, a large number of trees is preferred. No significant correlations can be seen between intrinsic stability and other factors. Finally, the magnitude of intrinsic stability is always smaller than that of traditional stability.
  
  
  CONCLUSIONS
  First, the prevalence of intrinsic stability of VIMs demonstrates that the instability of VIMs not only comes from data perturbations or parameter variations, but also stems from the intrinsic randomness of VIMs. This finding gives a better understanding of VIM stability, and may help reduce the instability of VIMs. Second, by investigating the potential factors of intrinsic stability, users would be more aware of the risks and hence more careful when using VIMs, especially on high-dimensional, small-sample and high complexity datasets.
authors: College of Computer Science and Technology, Huaqiao University, Jimei Avenue, Xiamen, 361021, China. wanghuazhen@hqu.edu.cn.|Computer Learning Research Centre, Royal Holloway, University of London, Egham, Surrey, TW20 0EX, UK. wanghuazhen@hqu.edu.cn.
authors: Huazhen
authors: H
authors: Wang
authors: Automation Department, Xiamen University, Siming South Road, Xiamen, 361005, China. yang@xmu.edu.cn.
authors: Fan
authors: F
authors: Yang
authors: Computer Learning Research Centre, Royal Holloway, University of London, Egham, Surrey, TW20 0EX, UK. zhiyuan@cs.rhul.ac.uk.
authors: Zhiyuan
authors: Z
authors: Luo
mesh_terms: D000465:Algorithms; D016000:Cluster Analysis; D019295:Computational Biology / Q000379:methods*; D020869:Gene Expression Profiling / Q000379:methods*; D005787:Gene Frequency; D006801:Humans; D008957:Models, Genetic; D020641:Polymorphism, Single Nucleotide / Q000235:genetics*; D012984:Software