title: Examining Reliability and Validity of an Online Score (ALiEM AIR) for Rating Free Open Access Medical Education Resources.
abstract: Since 2014, Academic Life in Emergency Medicine (ALiEM) has used the Approved Instructional Resources (AIR) score to critically appraise online content. The primary goals of this study are to determine the interrater reliability (IRR) of the ALiEM AIR rating score and determine its correlation with expert educator gestalt. We also determine the minimum number of educator-raters needed to achieve acceptable reliability.
  
  
  
  Eight educators each rated 83 online educational posts with the ALiEM AIR scale. Items include accuracy, usage of evidence-based medicine, referencing, utility, and the Best Evidence in Emergency Medicine rating score. A generalizability study was conducted to determine IRR and rating variance contributions of facets such as rater, blogs, posts, and topic. A randomized selection of 40 blog posts previously rated through ALiEM AIR was then rated again by a blinded group of expert medical educators according to their gestalt. Their gestalt impression was subsequently correlated with the ALiEM AIR score.
  
  
  
  The IRR for the ALiEM AIR rating scale was 0.81 during the 6-month pilot period. Decision studies showed that at least 9 raters were required to achieve this reliability. Spearman correlations between mean AIR score and the mean expert gestalt ratings were 0.40 for recommendation for learners and 0.35 for their colleagues.
  
  
  
  The ALiEM AIR scale is a moderately to highly reliable, 5-question tool when used by medical educators for rating online resources. The score displays a fair correlation with expert educator gestalt in regard to the quality of the resources. The score displays a fair correlation with educator gestalt.
authors.affiliation: Division of Emergency Medicine, McMaster University, Hamilton, ON, Canada; Academic Life in Emergency Medicine and the MedEdLIFE Research Collaborative. Electronic address: teresa.chan@medportal.ca.
authors.forename: Teresa Man-Yee
authors.initials: TM
authors.lastname: Chan
authors.affiliation: Department of Emergency Medicine, University of California, Olive View, CA, and Academic Life in Emergency Medicine.
authors.forename: Andrew
authors.initials: A
authors.lastname: Grock
authors.affiliation: Regions Hospital and the Department of Emergency Medicine, University of Minnesota.
authors.forename: Michael
authors.initials: M
authors.lastname: Paddock
authors.affiliation: Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, and the Wilson Centre for Health Professions Education, University Health Network, Toronto, ON, Canada.
authors.forename: Kulamakan
authors.initials: K
authors.lastname: Kulasegaram
authors.affiliation: Department of Emergency Medicine, Oregon Health & Science University, Portland, OR.
authors.forename: Lalena M
authors.initials: LM
authors.lastname: Yarris
authors.affiliation: Academic Life in Emergency Medicine and the MedEdLIFE Research Collaborative; Department of Emergency Medicine, University of California, San Francisco, CA.
authors.forename: Michelle
authors.initials: M
authors.lastname: Lin
mesh_terms: D022126:Access to Information; D020375:Education, Distance / Q000592:standards; D004502:Education, Medical, Continuing / Q000379:methods / Q000592:standards*; D004635:Emergency Medicine / Q000193:education; D006801:Humans; D015203:Reproducibility of Results