title: Automated confidence ranked classification of randomized controlled trial articles: an aid to evidence-based medicine.
abstract: OBJECTIVE
  For many literature review tasks, including systematic review (SR) and other aspects of evidence-based medicine, it is important to know whether an article describes a randomized controlled trial (RCT). Current manual annotation is not complete or flexible enough for the SR process. In this work, highly accurate machine learning predictive models were built that include confidence predictions of whether an article is an RCT.
  
  
  METHODS
  The LibSVM classifier was used with forward selection of potential feature sets on a large human-related subset of MEDLINE to create a classification model requiring only the citation, abstract, and MeSH terms for each article.
  
  
  RESULTS
  The model achieved an area under the receiver operating characteristic curve of 0.973 and mean squared error of 0.013 on the held out year 2011 data. Accurate confidence estimates were confirmed on a manually reviewed set of test articles. A second model not requiring MeSH terms was also created, and performs almost as well.
  
  
  CONCLUSIONS
  Both models accurately rank and predict article RCT confidence. Using the model and the manually reviewed samples, it is estimated that about 8000 (3%) additional RCTs can be identified in MEDLINE, and that 5% of articles tagged as RCTs in Medline may not be identified.
  
  
  CONCLUSIONS
  Retagging human-related studies with a continuously valued RCT confidence is potentially more useful for article ranking and review than a simple yes/no prediction. The automated RCT tagging tool should offer significant savings of time and effort during the process of writing SRs, and is a key component of a multistep text mining pipeline that we are building to streamline SR workflow. In addition, the model may be useful for identifying errors in MEDLINE publication types. The RCT confidence predictions described here have been made available to users as a web service with a user query form front end at: http://arrowsmith.psych.uic.edu/cgi-bin/arrowsmith_uic/RCT_Tagger.cgi.
authors: Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR 97239 USA cohenaa@ohsu.edu.
authors: Aaron M
authors: AM
authors: Cohen
authors: Department of Psychiatry, University of Illinois at Chicago, Chicago, IL 60612 USA.
authors: Neil R
authors: NR
authors: Smalheiser
authors: Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, Portland, OR 97239 USA.
authors: Marian S
authors: MS
authors: McDonagh
authors: Department of Computer Science, University of Illinois at Chicago, Chicago, IL 60612 USA.
authors: Clement
authors: C
authors: Yu
authors: Division of Psychiatry, University of Nottingham, Nottingham, UK.
authors: Clive E
authors: CE
authors: Adams
authors: Department of Psychiatry, University of Illinois at Chicago, Chicago, IL 60612 USA.
authors: John M
authors: JM
authors: Davis
authors: Department of Computer Science, University of Illinois at Chicago, Chicago, IL 60612 USA.
authors: Philip S
authors: PS
authors: Yu
mesh_terms: D001185:Artificial Intelligence; D019317:Evidence-Based Medicine; D006801:Humans; D016247:Information Storage and Retrieval / Q000379:methods*; D016239:MEDLINE; D012372:ROC Curve; D016032:Randomized Controlled Trials as Topic; D012196:Review Literature as Topic; D060388:Support Vector Machine
keywords: Evidence-Based Medicine; Information Retrieval; Natural Language Processing; Randomized Controlled Trials as Topic; Support Vector Machines; Systematic Reviews
references: Health Inf Sci Syst. 2014 Jan 10;2:1
references: 25825665
references: BMC Health Serv Res. 2002;2:3
references: 11882257
references: Cochrane Database Syst Rev. 2013;6:CD005440
references: 23813477
references: ACP J Club. 2005 Jan-Feb;142(1):A8-9
references: 15656539
references: J Am Med Inform Assoc. 2009 Jan-Feb;16(1):25-31
references: 18952929
references: BMJ. 2012;344:d7501
references: 22214757
references: J Clin Epidemiol. 2008 Aug;61(8):755-62
references: 18586179
references: Cochrane Database Syst Rev. 2010;(12):CD001867
references: 21154349
references: BMC Med Res Methodol. 2012;12:140
references: 22970848
references: Bioinformatics. 2009 Dec 1;25(23):3174-80
references: 19783830
references: J Am Med Inform Assoc. 2006 Mar-Apr;13(2):206-19
references: 16357352
references: Artif Intell Med. 2011 Jan;51(1):17-25
references: 21084178
references: J Am Med Inform Assoc. 2005 Mar-Apr;12(2):207-16
references: 15561789
references: BMJ. 1996 Jan 13;312(7023):71-2
references: 8555924
references: BMC Med Inform Decis Mak. 2012;12:33
references: 22515596
references: Methods. 2015 Mar;74:65-70
references: 25461812
references: Cochrane Database Syst Rev. 2011;(4):CD000978
references: 21491378
references: AMIA Annu Symp Proc. 2008;:121-5
references: 18998798
references: PLoS One. 2014;9(1):e86277
references: 24475099
references: Int J Epidemiol. 2002 Feb;31(1):150-3
references: 11914311
references: Cochrane Database Syst Rev. 2008;(1):MR000023
references: 18254126
references: Cochrane Database Syst Rev. 2010;(4):CD008189
references: 20393968
references: Int J Evid Based Healthc. 2012 Dec;10(4):397-410
references: 23173665
references: J Am Med Inform Assoc. 2013 Mar-Apr;20(2):363-8
references: 23019242
references: BMC Med Inform Decis Mak. 2009;9:10
references: 19208256
references: AMIA Annu Symp Proc. 2006;:161-5
references: 17238323
references: AMIA Annu Symp Proc. 2010;2010:121-5
references: 21346953
references: J Am Med Inform Assoc. 2010 Jul-Aug;17(4):446-53
references: 20595313
references: Artif Intell Med. 2012 Jul;55(3):197-207
references: 22677493
references: J Biomed Inform. 2009 Oct;42(5):801-13
references: 19022398
references: Bull Med Libr Assoc. 1983 Apr;71(2):176-83
references: 6344946
references: AMIA Annu Symp Proc. 2013;2013:379-87
references: 24551345
references: J Am Med Inform Assoc. 2006 Jul-Aug;13(4):446-55
references: 16622165
references: Syst Rev. 2012;1:10
references: 22587960
references: AMIA Annu Symp Proc. 2008;:151-5
references: 18998835
references: Database (Oxford). 2014;2014:bat086
references: 24434031
references: Qual Health Res. 2012 Oct;22(10):1435-43
references: 22829486