title: Individual Gestalt Is Unreliable for the Evaluation of Quality in Medical Education Blogs: A METRIQ Study.
abstract: OBJECTIVE
  Open educational resources such as blogs are increasingly used for medical education. Gestalt is generally the evaluation method used for these resources; however, little information has been published on it. We aim to evaluate the reliability of gestalt in the assessment of emergency medicine blogs.
  
  
  METHODS
  We identified 60 English-language emergency medicine Web sites that posted clinically oriented blogs between January 1, 2016, and February 24, 2016. Ten Web sites were selected with a random-number generator. Medical students, emergency medicine residents, and emergency medicine attending physicians evaluated the 2 most recent clinical blog posts from each site for quality, using a 7-point Likert scale. The mean gestalt scores of each blog post were compared between groups with Pearson's correlations. Single and average measure intraclass correlation coefficients were calculated within groups. A generalizability study evaluated variance within gestalt and a decision study calculated the number of raters required to reliably (>0.8) estimate quality.
  
  
  RESULTS
  One hundred twenty-one medical students, 88 residents, and 100 attending physicians (93.6% of enrolled participants) evaluated all 20 blog posts. Single-measure intraclass correlation coefficients within groups were fair to poor (0.36 to 0.40). Average-measure intraclass correlation coefficients were more reliable (0.811 to 0.840). Mean gestalt ratings by attending physicians correlated strongly with those by medical students (r=0.92) and residents (r=0.99). The generalizability coefficient was 0.91 for the complete data set. The decision study found that 42 gestalt ratings were required to reliably evaluate quality (>0.8).
  
  
  CONCLUSIONS
  The mean gestalt quality ratings of blog posts between medical students, residents, and attending physicians correlate strongly, but individual ratings are unreliable. With sufficient raters, mean gestalt ratings provide a community standard for assessment.
authors.affiliation: Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada; Health Professions Education Program, Massachusetts General Hospital Institute of Health Professions, Boston, MA. Electronic address: brent.thoma@usask.ca.
authors.forename: Brent
authors.initials: B
authors.lastname: Thoma
authors.affiliation: Centre for Education Research & Innovation, University of Western Ontario, London, Ontario, Canada.
authors.forename: Stefanie S
authors.initials: SS
authors.lastname: Sebok-Syer
authors.affiliation: Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada.
authors.forename: Keeth
authors.initials: K
authors.lastname: Krishnan
authors.affiliation: Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada.
authors.forename: Marshall
authors.initials: M
authors.lastname: Siemens
authors.affiliation: Department of Emergency Medicine, Northwestern University, Chicago, IL.
authors.forename: N Seth
authors.initials: NS
authors.lastname: Trueger
authors.affiliation: Department of Emergency Medicine, University of Alberta, Edmonton, Alberta, Canada.
authors.forename: Isabelle
authors.initials: I
authors.lastname: Colmers-Gray
authors.affiliation: Department of Emergency Medicine, University of Saskatchewan, Saskatoon, Saskatchewan, Canada.
authors.forename: Rob
authors.initials: R
authors.lastname: Woods
authors.affiliation: Health Professions Education Program, Massachusetts General Hospital Institute of Health Professions, Boston, MA.
authors.forename: Emil
authors.initials: E
authors.lastname: Petrusa
authors.affiliation: Division of Emergency Medicine, Department of Medicine, McMaster University, Hamilton, Ontario, Canada.
authors.forename: Teresa
authors.initials: T
authors.lastname: Chan
mesh_terms: D000328:Adult; D057218:Blogging / Q000592:standards* / Q000639:trends; D002983:Clinical Competence; D004501:Education, Medical / Q000379:methods / Q000592:standards*; D004521:Educational Measurement / Q000379:methods*; D004635:Emergency Medicine / Q000193:education*; D005260:Female; D005863:Gestalt Theory; D006801:Humans; D007396:Internship and Residency; D008297:Male; D015203:Reproducibility of Results; D061108:Social Media / Q000706:statistics & numerical data; D013337:Students, Medical