title: Syntactic parsing of clinical text: guideline and corpus development with handling ill-formed sentences.
abstract: OBJECTIVE
  To develop, evaluate, and share: (1) syntactic parsing guidelines for clinical text, with a new approach to handling ill-formed sentences; and (2) a clinical Treebank annotated according to the guidelines. To document the process and findings for readers with similar interest.
  
  
  METHODS
  Using random samples from a shared natural language processing challenge dataset, we developed a handbook of domain-customized syntactic parsing guidelines based on iterative annotation and adjudication between two institutions. Special considerations were incorporated into the guidelines for handling ill-formed sentences, which are common in clinical text. Intra- and inter-annotator agreement rates were used to evaluate consistency in following the guidelines. Quantitative and qualitative properties of the annotated Treebank, as well as its use to retrain a statistical parser, were reported.
  
  
  RESULTS
  A supplement to the Penn Treebank II guidelines was developed for annotating clinical sentences. After three iterations of annotation and adjudication on 450 sentences, the annotators reached an F-measure agreement rate of 0.930 (while intra-annotator rate was 0.948) on a final independent set. A total of 1100 sentences from progress notes were annotated that demonstrated domain-specific linguistic features. A statistical parser retrained with combined general English (mainly news text) annotations and our annotations achieved an accuracy of 0.811 (higher than models trained purely with either general or clinical sentences alone). Both the guidelines and syntactic annotations are made available at https://sourceforge.net/projects/medicaltreebank.
  
  
  CONCLUSIONS
  We developed guidelines for parsing clinical text and annotated a corpus accordingly. The high intra- and inter-annotator agreement rates showed decent consistency in following the guidelines. The corpus was shown to be useful in retraining a statistical parser that achieved moderate accuracy.
authors.affiliation: Medical Informatics, Kaiser Permanente Southern California, Pasadena, California, USA.
authors.forename: Jung-wei
authors.initials: JW
authors.lastname: Fan
authors.forename: Elly W
authors.initials: EW
authors.lastname: Yang
authors.forename: Min
authors.initials: M
authors.lastname: Jiang
authors.forename: Rashmi
authors.initials: R
authors.lastname: Prasad
authors.forename: Richard M
authors.initials: RM
authors.lastname: Loomis
authors.forename: Daniel S
authors.initials: DS
authors.lastname: Zisook
authors.forename: Josh C
authors.initials: JC
authors.lastname: Denny
authors.forename: Hua
authors.initials: H
authors.lastname: Xu
authors.forename: Yang
authors.initials: Y
authors.lastname: Huang
mesh_terms: D057286:Electronic Health Records; D017408:Guidelines as Topic; D008037:Linguistics; D009323:Natural Language Processing
keywords: annotation guidelines; corpus development; natural language processing; syntactic parsing
references.citation: Int J Med Inform. 2002 Dec 4;67(1-3):63-74
references.pmid: 12460632
references.citation: J Am Med Inform Assoc. 2013 Sep-Oct;20(5):922-30
references.pmid: 23355458
references.citation: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):552-6
references.pmid: 21685143
references.citation: IEEE Trans Inf Technol Biomed. 2007 Jul;11(4):364-75
references.pmid: 17674619
references.citation: J Am Med Inform Assoc. 2009 Jul-Aug;16(4):561-70
references.pmid: 19390096
references.citation: J Am Med Inform Assoc. 2005 May-Jun;12(3):275-85
references.pmid: 15684131
references.citation: J Am Med Inform Assoc. 2007 May-Jun;14(3):304-11
references.pmid: 17329723
references.citation: Methods Inf Med. 1996 Dec;35(4-5):285-301
references.pmid: 9019092
references.citation: J Biomed Inform. 2011 Oct;44(5):805-14
references.pmid: 21549857
references.citation: J Biomed Inform. 2009 Oct;42(5):790-800
references.pmid: 19166975
references.citation: J Am Med Inform Assoc. 1994 Mar-Apr;1(2):161-74
references.pmid: 7719797
references.citation: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):544-51
references.pmid: 21846786
references.citation: AMIA Annu Symp Proc. 2011;2011:382-91
references.pmid: 22195091
references.citation: J Am Med Inform Assoc. 2012 Jul-Aug;19(4):660-7
references.pmid: 22298565
references.citation: J Am Med Inform Assoc. 2011 Sep-Oct;18(5):540-3
references.pmid: 21846785
references.citation: Bioinformatics. 2009 Feb 1;25(3):394-400
references.pmid: 19073593
references.citation: Yearb Med Inform. 2008;:128-44
references.pmid: 18660887
references.citation: Proc Natl Acad Sci U S A. 2005 Aug 16;102(33):11629-34
references.pmid: 16087885
references.citation: J Am Med Inform Assoc. 2010 Sep-Oct;17(5):514-8
references.pmid: 20819854
references.citation: J Biomed Inform. 2002 Aug;35(4):222-35
references.pmid: 12755517
references.citation: J Am Med Inform Assoc. 2005 May-Jun;12(3):296-8
references.pmid: 15684123
references.citation: J Biomed Inform. 2012 Aug;45(4):763-71
references.pmid: 22326800