title: Evaluating temporal relations in clinical text: 2012 i2b2 Challenge.
abstract: BACKGROUND
  The Sixth Informatics for Integrating Biology and the Bedside (i2b2) Natural Language Processing Challenge for Clinical Records focused on the temporal relations in clinical narratives. The organizers provided the research community with a corpus of discharge summaries annotated with temporal information, to be used for the development and evaluation of temporal reasoning systems. 18 teams from around the world participated in the challenge. During the workshop, participating teams presented comprehensive reviews and analysis of their systems, and outlined future research directions suggested by the challenge contributions.
  
  
  METHODS
  The challenge evaluated systems on the information extraction tasks that targeted: (1) clinically significant events, including both clinical concepts such as problems, tests, treatments, and clinical departments, and events relevant to the patient's clinical timeline, such as admissions, transfers between departments, etc; (2) temporal expressions, referring to the dates, times, durations, or frequencies phrases in the clinical text. The values of the extracted temporal expressions had to be normalized to an ISO specification standard; and (3) temporal relations, between the clinical events and temporal expressions. Participants determined pairs of events and temporal expressions that exhibited a temporal relation, and identified the temporal relation between them.
  
  
  RESULTS
  For event detection, statistical machine learning (ML) methods consistently showed superior performance. While ML and rule based methods seemed to detect temporal expressions equally well, the best systems overwhelmingly adopted a rule based approach for value normalization. For temporal relation classification, the systems using hybrid approaches that combined ML and heuristics based methods produced the best results.
authors.affiliation: Department of Informatics, University at Albany, SUNY, Albany, New York, USA.
authors.forename: Weiyi
authors.initials: W
authors.lastname: Sun
authors.forename: Anna
authors.initials: A
authors.lastname: Rumshisky
authors.forename: Ozlem
authors.initials: O
authors.lastname: Uzuner
mesh_terms: D001185:Artificial Intelligence; D057286:Electronic Health Records; D006801:Humans; D009323:Natural Language Processing; D063886:Patient Discharge Summaries; D013995:Time; D057170:Translational Research, Biomedical
keywords: clinical language processing; medical language processing; natural language processing; sharedtask challenges; temporal reasoning
references.citation: J Biomed Inform. 2007 Apr;40(2):183-202
references.pmid: 17317332
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):820-7
references.pmid: 23571851
references.citation: J Am Med Inform Assoc. 2009 Jul-Aug;16(4):561-70
references.pmid: 19390096
references.citation: J Am Med Inform Assoc. 2010 Sep-Oct;17(5):507-13
references.pmid: 20819853
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):849-58
references.pmid: 23467472
references.citation: Proc AMIA Symp. 2001;:17-21
references.pmid: 11825149
references.citation: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):552-6
references.pmid: 21685143
references.citation: AMIA Annu Symp Proc. 2009 Nov 14;2009:568-72
references.pmid: 20351919
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):859-66
references.pmid: 23605114
references.citation: J Biomed Inform. 2013 Dec;46 Suppl:S54-S62
references.pmid: 24060600
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):867-75
references.pmid: 23686936
references.citation: Nucleic Acids Res. 2004 Jan 1;32(Database issue):D267-70
references.pmid: 14681409
references.citation: Yearb Med Inform. 2008;:128-44
references.pmid: 18660887
references.citation: J Biomed Inform. 2006 Aug;39(4):424-39
references.pmid: 16169282
references.citation: J Biomed Inform. 2013 Dec;46 Suppl:S20-S28
references.pmid: 23911344
references.citation: J Biomed Inform. 2013 Dec;46 Suppl:S40-S47
references.pmid: 24212118
references.citation: J Am Med Inform Assoc. 2008 Jan-Feb;15(1):14-24
references.pmid: 17947624
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):828-35
references.pmid: 23571849
references.citation: J Am Med Inform Assoc. 2007 Sep-Oct;14(5):550-63
references.pmid: 17600094
references.citation: J Am Med Inform Assoc. 2010 Sep-Oct;17(5):514-8
references.pmid: 20819854
references.citation: J Biomed Inform. 2013 Dec;46 Suppl:S29-S39
references.pmid: 23954311
references.citation: AMIA Annu Symp Proc. 2010 Nov 13;2010:787-91
references.pmid: 21347086
references.citation: J Biomed Inform. 2013 Dec;46 Suppl:S5-S12
references.pmid: 23872518
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):843-8
references.pmid: 23523875
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):836-42
references.pmid: 23558168
references.citation: J Am Med Inform Assoc. 2012 Sep-Oct;19(5):786-91
references.pmid: 22366294