title: Annotating risk factors for heart disease in clinical narratives for diabetic patients.
abstract: The 2014 i2b2/UTHealth natural language processing shared task featured a track focused on identifying risk factors for heart disease (specifically, Cardiac Artery Disease) in clinical narratives. For this track, we used a "light" annotation paradigm to annotate a set of 1304 longitudinal medical records describing 296 patients for risk factors and the times they were present. We designed the annotation task for this track with the goal of balancing annotation load and time with quality, so as to generate a gold standard corpus that can benefit a clinically-relevant task. We applied light annotation procedures and determined the gold standard using majority voting. On average, the agreement of annotators with the gold standard was above 0.95, indicating high reliability. The resulting document-level annotations generated for each record in each longitudinal EMR in this corpus provide information that can support studies of progression of heart disease risk factors in the included patients over time. These annotations were used in the Risk Factor track of the 2014 i2b2/UTHealth shared task. Participating systems achieved a mean micro-averaged F1 measure of 0.815 and a maximum F1 measure of 0.928 for identifying these risk factors in patient records.
authors.affiliation: School of Library and Information Science, Simmons College, Boston, MA, USA. Electronic address: stubbs@simmons.edu.
authors.forename: Amber
authors.initials: A
authors.lastname: Stubbs
authors.affiliation: Department of Information Studies, State University of New York at Albany, Albany, NY, USA.
authors.forename: Özlem
authors.initials: Ö
authors.lastname: Uzuner
mesh_terms: D000368:Aged; D001900:Boston / Q000453:epidemiology; D015331:Cohort Studies; D015897:Comorbidity; D016494:Computer Security; D003219:Confidentiality; D003324:Coronary Artery Disease / Q000175:diagnosis / Q000453:epidemiology*; D057225:Data Mining / Q000379:methods; D048909:Diabetes Complications / Q000175:diagnosis / Q000453:epidemiology*; D004282:Documentation / Q000379:methods*; D057286:Electronic Health Records / Q000458:organization & administration*; D005260:Female; D006801:Humans; D015994:Incidence; D008137:Longitudinal Studies; D008297:Male; D008875:Middle Aged; D033262:Narration; D009323:Natural Language Processing; D009518:New York / Q000453:epidemiology; D010363:Pattern Recognition, Automated / Q000379:methods; D018570:Risk Assessment / Q000379:methods; D018875:Vocabulary, Controlled
keywords: Annotation; Medical records; Natural language processing