title: The effect of sample size and disease prevalence on supervised machine learning of narrative data.
abstract: This paper examines the independent effects of outcome prevalence and training sample sizes on inductive learning performance. We trained 3 inductive learning algorithms (MC4, IB, and Naïve-Bayes) on 60 simulated datasets of parsed radiology text reports labeled with 6 disease states. Data sets were constructed to define positive outcome states at 4 prevalence rates (1, 5, 10, 25, and 50%) in training set sizes of 200 and 2,000 cases. We found that the effect of outcome prevalence is significant when outcome classes drop below 10% of cases. The effect appeared independent of sample size, induction algorithm used, or class label. Work is needed to identify methods of improving classifier performance when output classes are rare.
authors.affiliation: Department of Medical Informatics, Columbia University, New York, NY, USA.
authors.forename: Lawrence K
authors.initials: LK
authors.lastname: McKnight
authors.forename: Adam
authors.initials: A
authors.lastname: Wilcox
authors.forename: George
authors.initials: G
authors.lastname: Hripcsak
mesh_terms: D000465:Algorithms; D001185:Artificial Intelligence; D006801:Humans; D008168:Lung / Q000000981:diagnostic imaging*; D008171:Lung Diseases / Q000000981:diagnostic imaging; D011859:Radiography; D018401:Sample Size