title: Identifying medical terms in patient-authored text: a crowdsourcing-based approach.
abstract: OBJECTIVE
  As people increasingly engage in online health-seeking behavior and contribute to health-oriented websites, the volume of medical text authored by patients and other medical novices grows rapidly. However, we lack an effective method for automatically identifying medical terms in patient-authored text (PAT). We demonstrate that crowdsourcing PAT medical term identification tasks to non-experts is a viable method for creating large, accurately-labeled PAT datasets; moreover, such datasets can be used to train classifiers that outperform existing medical term identification tools.
  
  
  METHODS
  To evaluate the viability of using non-expert crowds to label PAT, we compare expert (registered nurses) and non-expert (Amazon Mechanical Turk workers; Turkers) responses to a PAT medical term identification task. Next, we build a crowd-labeled dataset comprising 10 000 sentences from MedHelp. We train two models on this dataset and evaluate their performance, as well as that of MetaMap, Open Biomedical Annotator (OBA), and NaCTeM's TerMINE, against two gold standard datasets: one from MedHelp and the other from CureTogether.
  
  
  RESULTS
  When aggregated according to a corroborative voting policy, Turker responses predict expert responses with an F1 score of 84%. A conditional random field (CRF) trained on 10 000 crowd-labeled MedHelp sentences achieves an F1 score of 78% against the CureTogether gold standard, widely outperforming OBA (47%), TerMINE (43%), and MetaMap (39%). A failure analysis of the CRF suggests that misclassified terms are likely to be either generic or rare.
  
  
  CONCLUSIONS
  Our results show that combining statistical models sensitive to sentence-level context with crowd-labeled data is a scalable and effective technique for automatically identifying medical terms in PAT.
authors: Department of Computer Science, Stanford University, Stanford, California, USA.
authors: Diana Lynn
authors: DL
authors: MacLean
authors: Jeffrey
authors: J
authors: Heer
mesh_terms: D054626:Consumer Health Information; D063045:Crowdsourcing; D016208:Databases, Factual; D006801:Humans; D016247:Information Storage and Retrieval; D009726:Nurses; D009626:Terminology as Topic
keywords: crowdsourcing; medical term extraction; online health forums; text mining
references: AMIA Annu Symp Proc. 2003;:529-33
references: 14728229
references: J Med Internet Res. 2007 Feb 28;9(1):e4
references: 17478413
references: Stud Health Technol Inform. 2004;107(Pt 1):487-91
references: 15360860
references: AMIA Annu Symp Proc. 2008 Nov 06;:682-6
references: 18999004
references: Summit Transl Bioinform. 2009 Mar 01;2009:56-60
references: 21347171
references: J Am Med Inform Assoc. 2008 Jul-Aug;15(4):496-505
references: 18436906
references: Nature. 2009 Feb 19;457(7232):1012-4
references: 19020500
references: J Am Med Inform Assoc. 2006 Jan-Feb;13(1):24-9
references: 16221948
references: Artif Intell Med. 2005 Feb;33(2):125-37
references: 15811781
references: J Med Internet Res. 2007 Mar 14;9(1):e5
references: 17478414
references: J Biomed Inform. 2011 Dec;44 Suppl 1:S39-S43
references: 21420508
references: J Med Internet Res. 2011 Jan 28;13(1):e15
references: 21278049
references: BMC Cancer. 2008 Jul 23;8:203
references: 18651949
references: J Biomed Inform. 2003 Aug-Oct;36(4-5):334-41
references: 14643729
references: Proc AMIA Symp. 2001;:17-21
references: 11825149
references: Proc AMIA Symp. 1999;:107-11
references: 10566330
references: J Fam Pract. 1987 Aug;25(2):176-8
references: 3612043
references: Methods Inf Med. 2002;41(4):289-98
references: 12425240
references: J Am Med Inform Assoc. 2010 May-Jun;17(3):229-36
references: 20442139
references: J Med Internet Res. 2011 May 17;13(2):e37
references: 21586386
references: J Am Med Inform Assoc. 2008 Mar-Apr;15(2):150-7
references: 18096908
references: Nat Biotechnol. 2011 May;29(5):411-4
references: 21516084