title: Consensus conference follow-up: inter-rater reliability assessment of the Best Evidence in Emergency Medicine (BEEM) rater scale, a medical literature rating tool for emergency physicians.
abstract: BACKGROUND
  Studies published in general and specialty medical journals have the potential to improve emergency medicine (EM) practice, but there can be delayed awareness of this evidence because emergency physicians (EPs) are unlikely to read most of these journals. Also, not all published studies are intended for or ready for clinical practice application. The authors developed "Best Evidence in Emergency Medicine" (BEEM) to ameliorate these problems by searching for, identifying, appraising, and translating potentially practice-changing studies for EPs. An initial step in the BEEM process is the BEEM rater scale, a novel tool for EPs to collectively evaluate the relative clinical relevance of EM-related studies found in more than 120 journals. The BEEM rater process was designed to serve as a clinical relevance filter to identify those studies with the greatest potential to affect EM practice. Therefore, only those studies identified by BEEM raters as having the highest clinical relevance are selected for the subsequent critical appraisal process and, if found methodologically sound, are promoted as the best evidence in EM.
  
  
  OBJECTIVE
  The primary objective was to measure inter-rater reliability (IRR) of the BEEM rater scale. Secondary objectives were to determine the minimum number of EP raters needed for the BEEM rater scale to achieve acceptable reliability and to compare performance of the scale against a previously published evidence rating system, the McMaster Online Rating of Evidence (MORE), in an EP population.
  
  
  METHODS
  The authors electronically distributed the title, conclusion, and a PubMed link for 23 recently published studies related to EM to a volunteer group of 134 EPs. The volunteers answered two demographic questions and rated the articles using one of two randomly assigned seven-point Likert scales, the BEEM rater scale (n = 68) or the MORE scale (n = 66), over two separate administrations. The IRR of each scale was measured using generalizability theory.
  
  
  RESULTS
  The IRR of the BEEM rater scale ranged between 0.90 (95% confidence interval [CI] = 0.86 to 0.93) to 0.92 (95% CI = 0.89 to 0.94) across administrations. Decision studies showed a minimum of 12 raters is required for acceptable reliability of the BEEM rater scale. The IRR of the MORE scale was 0.82 to 0.84.
  
  
  CONCLUSIONS
  The BEEM rater scale is a highly reliable, single-question tool for a small number of EPs to collectively rate the relative clinical relevance within the specialty of EM of recently published studies from a variety of medical journals. It compares favorably with the MORE system because it achieves a high IRR despite simply requiring raters to read each article's title and conclusion.
authors.affiliation: Department of Emergency Medicine, McMaster University, Hamilton, Ontario, Canada.
authors.forename: Andrew
authors.initials: A
authors.lastname: Worster
authors.forename: Kulamakan
authors.initials: K
authors.lastname: Kulasegaram
authors.forename: Christopher R
authors.initials: CR
authors.lastname: Carpenter
authors.forename: Teresa
authors.initials: T
authors.lastname: Vallera
authors.forename: Suneel
authors.initials: S
authors.lastname: Upadhye
authors.forename: Jonathan
authors.initials: J
authors.lastname: Sherbino
authors.forename: R Brian
authors.initials: RB
authors.lastname: Haynes
mesh_terms: D019985:Benchmarking / Q000458:organization & administration / Q000592:standards*; D003245:Consensus Statements as Topic; D004635:Emergency Medicine / Q000592:standards*; D055416:Evidence-Based Emergency Medicine / Q000458:organization & administration / Q000592:standards*; D006801:Humans; D015203:Reproducibility of Results
references.citation: J Emerg Med. 2009 Apr;36(3):317-22
references.pmid: 19097732
references.citation: Acad Emerg Med. 2003 May;10(5):417-22
references.pmid: 12720640
references.citation: J Emerg Med. 2010 May;38(4):518-23
references.pmid: 19846267
references.citation: J Emerg Med. 2009 Oct;37(3):313-8
references.pmid: 18164166
references.citation: Med Care. 1981 Dec;19(12):1211-20
references.pmid: 7339308
references.citation: J Emerg Med. 2009 Oct;37(3):319-27
references.pmid: 19427752
references.citation: JAMA. 2003 Apr 16;289(15):1969-75
references.pmid: 12697800
references.citation: J Contin Educ Health Prof. 2005 Summer;25(3):151-6
references.pmid: 16173049
references.citation: N Engl J Med. 2003 Jun 26;348(26):2635-45
references.pmid: 12826639
references.citation: BMC Med. 2004 Sep 06;2:33
references.pmid: 15350200
references.citation: Ann Emerg Med. 2009 Nov;54(5):748-50
references.pmid: 19185391
references.citation: JAMA. 2006 Apr 19;295(15):1801-8
references.pmid: 16622142
references.citation: JAMA. 2007 Sep 5;298(9):1010-22
references.pmid: 17785646
references.citation: Phys Ther. 1993 Jun;73(6):386-95; discussion 396-401
references.pmid: 8497513
references.citation: Adv Health Sci Educ Theory Pract. 2010 Dec;15(5):625-32
references.pmid: 20146096
references.citation: Med J Aust. 2004 Mar 15;180(S6):S57-60
references.pmid: 15012583
references.citation: Acad Emerg Med. 2007 Nov;14(11):1008-14
references.pmid: 17967963
references.citation: Emerg Med Clin North Am. 2006 Aug;24(3):785-95
references.pmid: 16877143
references.citation: Evid Based Med. 2009 Aug;14(4):121
references.pmid: 19648437
references.citation: Psychol Bull. 1979 Mar;86(2):420-8
references.pmid: 18839484
references.citation: BMJ. 2008 Mar 22;336(7645):655-7
references.pmid: 18292132
references.citation: Evid Based Med. 2009 Feb;14(1):25
references.pmid: 19181957
references.citation: Stat Med. 1990 Jun;9(6):645-56
references.pmid: 2218168
references.citation: CJEM. 2009 May;11(3):230-4
references.pmid: 19523271
references.citation: N Engl J Med. 1979 May 31;300(22):1242-5
references.pmid: 431682
references.citation: Acad Emerg Med. 2010 Oct;17 Suppl 2:S54-61
references.pmid: 21199085
references.citation: J Emerg Med. 2009 Feb;36(2):194-8; discussion 198-200
references.pmid: 18996671
references.citation: J Fam Pract. 1994 Nov;39(5):489-99
references.pmid: 7964548