title: Overfitting, generalization, and MSE in class probability estimation with high-dimensional data.
abstract: Accurate class probability estimation is important for medical decision making but is challenging, particularly when the number of candidate features exceeds the number of cases. Special methods have been developed for nonprobabilistic classification, but relatively little attention has been given to class probability estimation with numerous candidate variables. In this paper, we investigate overfitting in the development of regularized class probability estimators. We investigate the relation between overfitting and accurate class probability estimation in terms of mean square error. Using simulation studies based on real datasets, we found that some degree of overfitting can be desirable for reducing mean square error. We also introduce a mean square error decomposition for class probability estimation that helps clarify the relationship between overfitting and prediction accuracy.
authors: Biometric Research Branch, National Cancer Institute, 9609 Medical Center Dr, MSC 9735 Bethesda, MD 20892-9735, USA.
authors: Kyung In
authors: KI
authors: Kim
authors: Richard
authors: R
authors: Simon
mesh_terms: D001499:Bayes Theorem; D001699:Biometry / Q000379:methods*; D009369:Neoplasms / Q000235:genetics / Q000473:pathology; D020411:Oligonucleotide Array Sequence Analysis; D011336:Probability; D012044:Regression Analysis; D012107:Research Design; D059467:Transcriptome
keywords: Class probability estimation; Covariance penalty; High-dimensional data; Mean square error; Overfitting