title: The role of fine-grained annotations in supervised recognition of risk factors for heart disease from EHRs.
abstract: This paper describes a supervised machine learning approach for identifying heart disease risk factors in clinical text, and assessing the impact of annotation granularity and quality on the system's ability to recognize these risk factors. We utilize a series of support vector machine models in conjunction with manually built lexicons to classify triggers specific to each risk factor. The features used for classification were quite simple, utilizing only lexical information and ignoring higher-level linguistic information such as syntax and semantics. Instead, we incorporated high-quality data to train the models by annotating additional information on top of a standard corpus. Despite the relative simplicity of the system, it achieves the highest scores (micro- and macro-F1, and micro- and macro-recall) out of the 20 participants in the 2014 i2b2/UTHealth Shared Task. This system obtains a micro- (macro-) precision of 0.8951 (0.8965), recall of 0.9625 (0.9611), and F1-measure of 0.9276 (0.9277). Additionally, we perform a series of experiments to assess the value of the annotated data we created. These experiments show how manually-labeled negative annotations can improve information extraction performance, demonstrating the importance of high-quality, fine-grained natural language annotations.
authors.affiliation: Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA. Electronic address: kirk.roberts@nih.gov.
authors.forename: Kirk
authors.initials: K
authors.lastname: Roberts
authors.affiliation: Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
authors.forename: Sonya E
authors.initials: SE
authors.lastname: Shooshan
authors.affiliation: Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
authors.forename: Laritza
authors.initials: L
authors.lastname: Rodriguez
authors.affiliation: Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
authors.forename: Swapna
authors.initials: S
authors.lastname: Abhyankar
authors.affiliation: Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
authors.forename: Halil
authors.initials: H
authors.lastname: Kilicoglu
authors.affiliation: Lister Hill National Center for Biomedical Communications, National Library of Medicine, National Institutes of Health, Bethesda, MD, USA.
authors.forename: Dina
authors.initials: D
authors.lastname: Demner-Fushman
mesh_terms: D000368:Aged; D015331:Cohort Studies; D015897:Comorbidity; D016494:Computer Security; D003219:Confidentiality; D003324:Coronary Artery Disease / Q000175:diagnosis / Q000453:epidemiology*; D057225:Data Mining / Q000379:methods*; D048909:Diabetes Complications / Q000175:diagnosis / Q000453:epidemiology*; D057286:Electronic Health Records / Q000458:organization & administration*; D005260:Female; D006801:Humans; D015994:Incidence; D008137:Longitudinal Studies; D008297:Male; D008396:Maryland / Q000453:epidemiology; D008875:Middle Aged; D033262:Narration; D009323:Natural Language Processing; D010363:Pattern Recognition, Automated / Q000379:methods; D018570:Risk Assessment / Q000379:methods; D000069553:Supervised Machine Learning; D018875:Vocabulary, Controlled
keywords: Machine learning; Natural language annotation; Natural language processing