title: Characterization of the effectiveness of reporting lists of small feature sets relative to the accuracy of the prior biological knowledge.
abstract: When confronted with a small sample, feature-selection algorithms often fail to find good feature sets, a problem exacerbated for high-dimensional data and large feature sets. The problem is compounded by the fact that, if one obtains a feature set with a low error estimate, the estimate is unreliable because training-data-based error estimators typically perform poorly on small samples, exhibiting optimistic bias or high variance. One way around the problem is limit the number of features being considered, restrict features sets to sizes such that all feature sets can be examined by exhaustive search, and report a list of the best performing feature sets. If the list is short, then it greatly restricts the possible feature sets to be considered as candidates; however, one can expect the lowest error estimates obtained to be optimistically biased so that there may not be a close-to-optimal feature set on the list. This paper provides a power analysis of this methodology; in particular, it examines the kind of results one should expect to obtain relative to the length of the list and the number of discriminating features among those considered. Two measures are employed. The first is the probability that there is at least one feature set on the list whose true classification error is within some given tolerance of the best feature set and the second is the expected number of feature sets on the list whose true errors are within the given tolerance of the best feature set. These values are plotted as functions of the list length to generate power curves. The results show that, if the number of discriminating features is not too small-that is, the prior biological knowledge is not too poor-then one should expect, with high probability, to find good feature sets.
authors.affiliation: Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX 77843, USA.
authors.forename: Chen
authors.initials: C
authors.lastname: Zhao
authors.forename: Michael L
authors.initials: ML
authors.lastname: Bittner
authors.forename: Robert S
authors.initials: RS
authors.lastname: Chapkin
authors.forename: Edward R
authors.initials: ER
authors.lastname: Dougherty
references.citation: Cancer Res. 2003 Jan 1;63(1):60-6
references.pmid: 12517778
references.citation: J Natl Cancer Inst. 2004 Mar 17;96(6):434-42
references.pmid: 15026468
references.citation: Cancer Prev Res (Phila). 2009 Jun;2(6):590-7
references.pmid: 19470793
references.citation: Int J Oncol. 2003 Sep;23(3):617-25
references.pmid: 12888896
references.citation: JAMA. 2007 Apr 11;297(14):1551-61
references.pmid: 17426274
references.citation: Bioinformatics. 2006 Oct 1;22(19):2430-6
references.pmid: 16870934
references.citation: Bioinformatics. 2005 Apr 15;21(8):1509-15
references.pmid: 15572470
references.citation: Bioinformatics. 2005 Apr 1;21(7):1046-54
references.pmid: 15514003
references.citation: BMC Bioinformatics. 2005 Apr 13;6:97
references.pmid: 15826317
references.citation: EURASIP J Bioinform Syst Biol. 2007;:16354
references.pmid: 17713587
references.citation: Bioinformatics. 2005 Aug 1;21(15):3301-7
references.pmid: 15905277
references.citation: Brief Bioinform. 2008 Sep;9(5):392-403
references.pmid: 18562478
references.citation: EURASIP J Bioinform Syst Biol. 2007;:38473
references.pmid: 18288255
references.citation: Bioinformatics. 2007 Oct 1;23(19):2507-17
references.pmid: 17720704
references.citation: Mol Cancer Ther. 2002 Nov;1(13):1229-36
references.pmid: 12479704
references.citation: Bioinformatics. 2004 Feb 12;20(3):374-80
references.pmid: 14960464