title: Formative evaluation of the accuracy of a clinical decision support system for cervical cancer screening.
abstract: OBJECTIVE
  We previously developed and reported on a prototype clinical decision support system (CDSS) for cervical cancer screening. However, the system is complex as it is based on multiple guidelines and free-text processing. Therefore, the system is susceptible to failures. This report describes a formative evaluation of the system, which is a necessary step to ensure deployment readiness of the system.
  
  
  METHODS
  Care providers who are potential end-users of the CDSS were invited to provide their recommendations for a random set of patients that represented diverse decision scenarios. The recommendations of the care providers and those generated by the CDSS were compared. Mismatched recommendations were reviewed by two independent experts.
  
  
  RESULTS
  A total of 25 users participated in this study and provided recommendations for 175 cases. The CDSS had an accuracy of 87% and 12 types of CDSS errors were identified, which were mainly due to deficiencies in the system's guideline rules. When the deficiencies were rectified, the CDSS generated optimal recommendations for all failure cases, except one with incomplete documentation.
  
  
  CONCLUSIONS
  The crowd-sourcing approach for construction of the reference set, coupled with the expert review of mismatched recommendations, facilitated an effective evaluation and enhancement of the system, by identifying decision scenarios that were missed by the system's developers. The described methodology will be useful for other researchers who seek rapidly to evaluate and enhance the deployment readiness of complex decision support systems.
authors.affiliation: Division of Biomedical Statistics and Informatics, Mayo Clinic, Rochester, Minnesota 55901, USA. waghsk@gmail.com
authors.forename: Kavishwar Balwant
authors.initials: KB
authors.lastname: Wagholikar
authors.forename: Kathy L
authors.initials: KL
authors.lastname: MacLaughlin
authors.forename: Thomas M
authors.initials: TM
authors.lastname: Kastner
authors.forename: Petra M
authors.initials: PM
authors.lastname: Casey
authors.forename: Michael
authors.initials: M
authors.lastname: Henry
authors.forename: Robert A
authors.initials: RA
authors.lastname: Greenes
authors.forename: Hongfang
authors.initials: H
authors.lastname: Liu
authors.forename: Rajeev
authors.initials: R
authors.lastname: Chaudhry
mesh_terms: D057225:Data Mining; D020000:Decision Support Systems, Clinical; D055088:Early Detection of Cancer; D057286:Electronic Health Records; D005260:Female; D006801:Humans; D009323:Natural Language Processing; D017410:Practice Guidelines as Topic; D002583:Uterine Cervical Neoplasms / Q000175:diagnosis*
keywords: Crowdsourcing; Decision Support Systems, Clinical; Guideline Adherence; Uterine Cervical Neoplasms; Vaginal Smears; Validation Studies as Topic
references.citation: J Natl Cancer Inst. 1997 May 21;89(10):703-8
references.pmid: 9168185
references.citation: Arch Intern Med. 2010 Jun 14;170(11):977-85
references.pmid: 20548011
references.citation: J Am Med Inform Assoc. 2012 Sep-Oct;19(5):713-8
references.pmid: 22582202
references.citation: CA Cancer J Clin. 2002 Nov-Dec;52(6):342-62
references.pmid: 12469763
references.citation: Med Care. 1998 Mar;36(3):397-410
references.pmid: 9520963
references.citation: Cancer Epidemiol Biomarkers Prev. 2011 Dec;20(12):2471-9
references.pmid: 21976292
references.citation: J Natl Cancer Inst Monogr. 2010;2010(40):58-71
references.pmid: 20386054
references.citation: Ann Intern Med. 2012 Jun 19;156(12):880-91, W312
references.pmid: 22711081
references.citation: J Am Med Inform Assoc. 2008 May-Jun;15(3):290-6
references.pmid: 18308985
references.citation: Milbank Q. 2012 Mar;90(1):38-41
references.pmid: 22428691
references.citation: AMIA Jt Summits Transl Sci Proc. 2013 Mar 18;2013:269-73
references.pmid: 24303280
references.citation: Isr Med Assoc J. 2009 Jan;11(1):23-9
references.pmid: 19344008
references.citation: Prev Med. 2007 Aug-Sep;45(2-3):93-106
references.pmid: 17651792
references.citation: Stud Health Technol Inform. 2004;107(Pt 1):174-8
references.pmid: 15360798
references.citation: J Am Med Inform Assoc. 2012 Sep-Oct;19(5):833-9
references.pmid: 22542812
references.citation: Health Aff (Millwood). 2011 Dec;30(12):2310-7
references.pmid: 22147858
references.citation: Cochrane Database Syst Rev. 2011 Apr 13;(4):CD001035
references.pmid: 21491379
references.citation: Obstet Gynecol. 2010 Dec;116(6):1332-1340
references.pmid: 21099599
references.citation: Aust N Z J Public Health. 2003;27(3):273-81
references.pmid: 14705281
references.citation: Br J Gen Pract. 2010 Apr;60(573):e137-43
references.pmid: 20353659
references.citation: Arch Intern Med. 2009 Feb 23;169(4):364-71
references.pmid: 19237720
references.citation: Obstet Gynecol. 2011 Jul;118(1):4-13
references.pmid: 21691157
references.citation: CA Cancer J Clin. 2012 Jan-Feb;62(1):10-29
references.pmid: 22237781
references.citation: Am J Obstet Gynecol. 2007 Oct;197(4):346-55
references.pmid: 17904957
references.citation: Am J Obstet Gynecol. 2011 Nov;205(5):447.e1-8
references.pmid: 21840492
references.citation: Cochrane Database Syst Rev. 2011 May 11;(5):CD002834
references.pmid: 21563135
references.citation: Ann Intern Med. 2009 Nov 3;151(9):602-11
references.pmid: 19884621
references.citation: Comput Nurs. 2000 Nov-Dec;18(6):265-71
references.pmid: 11105400
references.citation: J Gen Intern Med. 2012 Jan;27(1):85-92
references.pmid: 21904945
references.citation: Obstet Gynecol. 2009 Dec;114(6):1409-1420
references.pmid: 20134296
references.citation: J Am Med Inform Assoc. 2011 May 1;18(3):243-50
references.pmid: 21486882
references.citation: Proc AMIA Symp. 2001;:12-6
references.pmid: 11825148
references.citation: J Med Libr Assoc. 2005 Oct;93(4 Suppl):S43-8
references.pmid: 16239957
references.citation: Ann Intern Med. 2012 Jul 3;157(1):29-43
references.pmid: 22751758
references.citation: Arch Intern Med. 2011 Feb 14;171(3):249-56
references.pmid: 20937917
references.citation: Ann Oncol. 2011 Dec;22(12):2675-2686
references.pmid: 21471563
references.citation: Int J Med Inform. 2010 Nov;79(11):772-7
references.pmid: 20951080
references.citation: J Am Med Inform Assoc. 2003 Jan-Feb;10(1):52-68
references.pmid: 12509357
references.citation: J Med Syst. 2012 Oct;36(5):3029-49
references.pmid: 21964969
references.citation: J Low Genit Tract Dis. 2012 Jul;16(3):175-204
references.pmid: 22418039
references.citation: J Am Med Inform Assoc. 2010 Jul-Aug;17(4):411-5
references.pmid: 20595308
references.citation: J Biomed Inform. 2009 Oct;42(5):760-72
references.pmid: 19683066
references.citation: Healthc Inform. 2012 Apr;29(4):18, 20
references.pmid: 22574398
references.citation: Aust N Z J Public Health. 2005 Jun;29(3):261-4
references.pmid: 15991775
references.citation: J Am Med Inform Assoc. 1996 Jan-Feb;3(1):42-55
references.pmid: 8750389
references.citation: J Reprod Med. 2007 Nov;52(11):995-1000
references.pmid: 18161396
references.citation: J Am Med Inform Assoc. 2011 May 1;18(3):327-34
references.pmid: 21422100
references.citation: J Am Med Inform Assoc. 2008 May-Jun;15(3):297-301
references.pmid: 18308984
references.citation: Gynecol Oncol. 2010 Jun;117(3):486-90
references.pmid: 20303156
references.citation: J Gen Intern Med. 2001 Aug;16(8):563-8
references.pmid: 11556935
references.citation: J Biomed Inform. 2008 Apr;41(2):387-92
references.pmid: 18029232
references.citation: J Gen Intern Med. 2003 May;18(5):357-63
references.pmid: 12795734
references.citation: BMJ Qual Saf. 2011 Nov;20(11):924-30
references.pmid: 21725046