Characterization of the effectiveness of reporting lists of small feature sets relative to the accuracy of the prior biological knowledge.
When confronted with a small sample, feature-selection algorithms often fail to find good feature sets, a problem exacerbated for high-dimensional data and large feature sets. The problem is compounded by the fact that, if one obtains a feature set with a low error estimate, the estimate is unreliable because training-data-based error estimators typically perform poorly on small samples, exhibiting optimistic bias or high variance. One way around the problem is limit the number of features being considered, restrict features sets to sizes such that all feature sets can be examined by exhaustive search, and report a list of the best performing feature sets. If the list is short, then it greatly restricts the possible feature sets to be considered as candidates; however, one can expect the lowest error estimates obtained to be optimistically biased so that there may not be a close-to-optimal feature set on the list. This paper provides a power analysis of this methodology; in particular, it examines the kind of results one should expect to obtain relative to the length of the list and the number of discriminating features among those considered. Two measures are employed. The first is the probability that there is at least one feature set on the list whose true classification error is within some given tolerance of the best feature set and the second is the expected number of feature sets on the list whose true errors are within the given tolerance of the best feature set. These values are plotted as functions of the list length to generate power curves. The results show that, if the number of discriminating features is not too small-that is, the prior biological knowledge is not too poor-then one should expect, with high probability, to find good feature sets.
Zhao
Chen
C
Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843, USA.
Bittner
Michael L
ML
Chapkin
Robert S
RS
Dougherty
Edward R
ER
Cancer Res. 2003 Jan 1;63(1):60-6
12517778
J Natl Cancer Inst. 2004 Mar 17;96(6):434-42
15026468
Cancer Prev Res (Phila). 2009 Jun;2(6):590-7
19470793
Int J Oncol. 2003 Sep;23(3):617-25
12888896
JAMA. 2007 Apr 11;297(14):1551-61
17426274
Bioinformatics. 2006 Oct 1;22(19):2430-6
16870934
Bioinformatics. 2005 Apr 15;21(8):1509-15
15572470
Bioinformatics. 2005 Apr 1;21(7):1046-54
15514003
BMC Bioinformatics. 2005 Apr 13;6:97
15826317
EURASIP J Bioinform Syst Biol. 2007;:16354
17713587
Bioinformatics. 2005 Aug 1;21(15):3301-7
15905277
Brief Bioinform. 2008 Sep;9(5):392-403
18562478
EURASIP J Bioinform Syst Biol. 2007;:38473
18288255
Bioinformatics. 2007 Oct 1;23(19):2507-17
17720704
Mol Cancer Ther. 2002 Nov;1(13):1229-36
12479704
Bioinformatics. 2004 Feb 12;20(3):374-80
14960464