title: Measurement of faculty anesthesiologists' quality of clinical supervision has greater reliability when controlling for the leniency of the rating anesthesia resident: a retrospective cohort study.
abstract: BACKGROUND
  Our department monitors the quality of anesthesiologists' clinical supervision and provides each anesthesiologist with periodic feedback. We hypothesized that greater differentiation among anesthesiologists' supervision scores could be obtained by adjusting for leniency of the rating resident.
  
  
  METHODS
  From July 1, 2013 to December 31, 2015, our department has utilized the de Oliveira Filho unidimensional nine-item supervision scale to assess the quality of clinical supervision provided by faculty as rated by residents. We examined all 13,664 ratings of the 97 anesthesiologists (ratees) by the 65 residents (raters). Testing for internal consistency among answers to questions (large Cronbach's alpha > 0.90) was performed to rule out that one or two questions accounted for leniency. Mixed-effects logistic regression was used to compare ratees while controlling for rater leniency vs using Student t tests without rater leniency.
  
  
  RESULTS
  The mean supervision scale score was calculated for each combination of the 65 raters and nine questions. The Cronbach's alpha was very large (0.977). The mean score was calculated for each of the 3,421 observed combinations of resident and anesthesiologist. The logits of the percentage of scores equal to the maximum value of 4.00 were normally distributed (residents, P = 0.24; anesthesiologists, P = 0.50). There were 20/97 anesthesiologists identified as significant outliers (13 with below average supervision scores and seven with better than average) using the mixed-effects logistic regression with rater leniency entered as a fixed effect but not by Student's t test. In contrast, there were three of 97 anesthesiologists identified as outliers (all three above average) using Student's t tests but not by logistic regression with leniency. The 20 vs 3 was significant (P < 0.001).
  
  
  CONCLUSIONS
  Use of logistic regression with leniency results in greater detection of anesthesiologists with significantly better (or worse) clinical supervision scores than use of Student's t tests (i.e., without adjustment for rater leniency).
authors.affiliation: Division of Management Consulting, University of Iowa, 200 Hawkins Drive, 6-JCP, Iowa City, IA, 52242, USA. Franklin-Dexter@UIowa.edu.|Department of Anesthesia, University of Iowa, Iowa City, IA, USA. Franklin-Dexter@UIowa.edu.
authors.forename: Franklin
authors.identifier: http://orcid.org/0000-0001-5897-2484
authors.initials: F
authors.lastname: Dexter
authors.affiliation: Department of Management Sciences, University of Iowa, Iowa City, IA, USA.
authors.forename: Johannes
authors.initials: J
authors.lastname: Ledolter
authors.affiliation: Department of Anesthesia, University of Iowa, Iowa City, IA, USA.
authors.forename: Bradley J
authors.initials: BJ
authors.lastname: Hindman
mesh_terms: D000072080:Anesthesiologists / Q000592:standards*; D000776:Anesthesiology / Q000193:education*; D002983:Clinical Competence; D015331:Cohort Studies; D005180:Faculty, Medical / Q000592:standards*; D006801:Humans; D007396:Internship and Residency; D016015:Logistic Models; D015203:Reproducibility of Results; D012189:Retrospective Studies