Derivation of Two Critical Appraisal Scores for Trainees to Evaluate Online Educational Resources: A METRIQ Study.
BACKGROUND
Online education resources (OERs), like blogs and podcasts, increasingly augment or replace traditional medical education resources such as textbooks and lectures. Trainees' ability to evaluate these resources is poor, and few quality assessment aids have been developed to assist them. This study aimed to derive a quality evaluation instrument for this purpose.


METHODS
We used a three-phase methodology. In Phase 1, a previously derived list of 151 OER quality indicators was reduced to 13 items using data from published consensus-building studies (of medical educators, expert podcasters, and expert bloggers) and subsequent evaluation by our team. In Phase 2, these 13 items were converted to seven-point Likert scales used by trainee raters (n=40) to evaluate 39 OERs. The reliability and usability of these 13 rating items was determined using responses from trainee raters, and top items were used to create two OER quality evaluation instruments. In Phase 3, these instruments were compared to an external certification process (the ALiEM AIR certification) and the gestalt evaluation of the same 39 blog posts by 20 faculty educators.


RESULTS
Two quality-evaluation instruments were derived with fair inter-rater reliability: the METRIQ-8 Score (Inter class correlation coefficient [ICC]=0.30, p<0.001) and the METRIQ-5 Score (ICC=0.22, p<0.001). Both scores, when calculated using the derivation data, correlated with educator gestalt (Pearson's r=0.35, p=0.03 and r=0.41, p<0.01, respectively) and were related to increased odds of receiving an ALiEM AIR certification (odds ratio=1.28, p=0.03; OR=1.5, p=0.004, respectively).


CONCLUSIONS
Two novel scoring instruments with adequate psychometric properties were derived to assist trainees in evaluating OER quality and correlated favourably with gestalt ratings of online educational resources by faculty educators. Further testing is needed to ensure these instruments are accurate when applied by trainees.
McMaster University, Department of Medicine, Division of Emergency Medicine, Hamilton, Ontario, Canada.
Teresa M
TM
Chan
University of Saskatchewan, Department of Emergency Medicine, Saskatoon, Saskatchewan, Canada.
Brent
B
Thoma
University of Toronto, Faculty of Medicine, Undergraduate Medical Education, Toronto, Ontario, Canada.
Keeth
K
Krishnan
University of California, San Francisco, School of Medicine, Department of Emergency Medicine, San Francisco, California.
Michelle
M
Lin
Washington University School of Medicine, St. Louis, Missouri.
Christopher R
CR
Carpenter
Mercer University School of Medicine, Department of Emergency Medicine, Department of Internal Medicine, Macon, Georgia.
Matt
M
Astin
University of Toronto Faculty of Medicine, Department of Family and Community Medicine, Toronto, Ontario, Canada; University Health Network, Department of Emergency Medicine, Wilson Centre for Health Professions Education, Toronto, Ontario, Canada.
Kulamakan
K
Kulasegaram
D002983:Clinical Competence; D003479:Curriculum / Q000639:trends*; D004501:Education, Medical / Q000592:standards*; D006801:Humans; D020407:Internet / Q000639:trends; D007396:Internship and Residency; D015203:Reproducibility of Results; D061108:Social Media / Q000706:statistics & numerical data; D011795:Surveys and Questionnaires