title: Challenging script concordance test reference standard by evidence: do judgments by emergency medicine consultants agree with likelihood ratios?
abstract: BACKGROUND
  We aimed to compare the clinical judgments of a reference panel of emergency medicine academic physicians against evidence-based likelihood ratios (LRs) regarding the diagnostic value of selected clinical and paraclinical findings in the context of a script concordance test (SCT).
  
  
  RESULTS
  A SCT with six scenarios and five questions per scenario was developed. Subsequently, 15 emergency medicine attending physicians (reference panel) took the test and their judgments regarding the diagnostic value of those findings for given diseases were recorded. The LRs of the same findings for the same diseases were extracted from a series of published systematic reviews. Then, the reference panel judgments were compared to evidence-based LRs. To investigate the test-retest reliability, five participants took the test one month later, and the correlation of their first and second judgments were quantified using Spearman rank-order coefficient. In 22 out of 30 (73.3%) findings, the expert judgments were significantly different from the LRs. The differences included overestimation (30%), underestimation (30%), and judging the diagnostic value in an opposite direction (13.3%). Moreover, the score of a hypothetical test-taker was calculated to be 21.73 out of 30 if his/her answers were based on evidence-based LRs. The test showed an acceptable test-retest reliability coefficient (Spearman coefficient: 0.83).
  
  
  CONCLUSIONS
  Although SCT is an interesting test to evaluate clinical decision-making in emergency medicine, our results raise concerns regarding whether the judgments of an expert panel are sufficiently valid as the reference standard for this test.
authors: Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran ; Program in Public Health, Department of Population Health and Disease Prevention, University of California Irvine, 653 E. Peltason Dr., Irvine 92697, CA, USA.
authors: Seyed-Foad
authors: SF
authors: Ahmadi
authors: Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran ; Klinik für Innere Medizin III, Kardiologie, Angiologie und Internistische Intensivmedizin, Universitätsklinikum des Saarlandes, Homburg/Saar 66421, Germany.
authors: Shahin
authors: S
authors: Khoshkish
authors: Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran.
authors: Kamran
authors: K
authors: Soltani-Arabshahi
authors: Department of Emergency Medicine, Iran University of Medical Sciences, Tehran 14496, Iran.
authors: Peyman
authors: P
authors: Hafezi-Moghadam
authors: Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran.
authors: Golara
authors: G
authors: Zahmatkesh
authors: Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran ; Department of Neurology, Saarland University Medical Center, Homburg/Saar 66421, Germany.
authors: Parisa
authors: P
authors: Heidari
authors: Kamyar Clinic, Tehran 51406, Iran.
authors: Davood
authors: D
authors: Baba-Beigloo
authors: Center for Educational Research in Medical Sciences, Iran University of Medical Sciences, Tehran 14496, Iran.
authors: Hamid R
authors: HR
authors: Baradaran
authors: Department of Emergency Medicine, School of Medicine, University of California Irvine, Irvine 92697, CA, USA.
authors: Shahram
authors: S
authors: Lotfipour
keywords: Clinical judgment; Decision-making; Diagnosis; Evidence-based medicine; Likelihood ratio; Script concordance test; Visual analog scales
references: Fam Med. 2003 Apr;35(4):257-60
references: 12729309
references: Teach Learn Med. 2010 Jul;22(3):180-6
references: 20563937
references: Can J Neurol Sci. 2009 May;36(3):326-31
references: 19534333
references: J Am Geriatr Soc. 2012 Oct;60(10):1946-50
references: 23036106
references: Adv Health Sci Educ Theory Pract. 2011 Dec;16(5):601-8
references: 21286807
references: Acad Emerg Med. 2011 Jun;18(6):627-34
references: 21676061
references: J Chronic Dis. 1987;40(12):1129-33
references: 3680471
references: BMC Med Inform Decis Mak. 2008 May 06;8:18
references: 18460199
references: Med Teach. 2011;33(6):472-7
references: 21609176
references: Acad Med. 2002 Oct;77(10):981-92
references: 12377672
references: Med Educ. 2010 Jan;44(1):94-100
references: 20078760
references: Med Educ. 2006 Sep;40(9):848-54
references: 16925634
references: Acta Paediatr. 2004 Jun;93(6):830-5
references: 15244235
references: Radiat Oncol. 2009 Feb 09;4:7
references: 19203358
references: Ann N Y Acad Sci. 1993 Dec 31;703:125-33; discussion 133-4
references: 8192290
references: Am J Surg. 2013 Feb;205(2):137-40
references: 23246286
references: Arch Otolaryngol Head Neck Surg. 2011 Aug;137(8):751-5
references: 21844407
references: N Engl J Med. 2006 Nov 23;355(21):2217-25
references: 17124019
references: Am J Obstet Gynecol. 2010 Sep;203(3):240.e1-6
references: 20494330
references: Ann Emerg Med. 2009 May;53(5):647-52
references: 18722694
references: Swiss Med Wkly. 2012 Oct 23;142:w13706
references: 23135902
references: Br Med J. 1975 May 31;2(5969):486-9
references: 1148666
references: Surg Innov. 2006 Jun;13(2):115-9
references: 17012152
references: Emerg Med J. 2014 Apr;31(4):313-6
references: 23539495
references: BMC Med Educ. 2012 May 09;12:29
references: 22571351
references: BMC Med Educ. 2013 Dec 13;13:166
references: 24330600
references: Am J Surg. 2012 Apr;203(4):530-4
references: 22450028
references: J Eur Acad Dermatol Venereol. 2011 Dec;25(12):1471-5
references: 22077872
references: Acad Emerg Med. 2010 May;17 (5):521-3
references: 20536807
references: Med Educ. 2013 Dec;47(12):1175-83
references: 24206151