title: Estimating the Reliability of Nontechnical Skills in Medical Teams.
abstract: OBJECTIVE
  To assess the reliability of measurement of assessments of nontechnical skills in medical teams.
  
  
  BACKGROUND
  There are several scales for assessing nontechnical skills in medical teams. In this critique, we describe the reliability of such assessments and discuss factors that may influence the obtained estimate of reliability for such scales.
  
  
  METHODS
  A review of the literature yielded 28 reliability estimates, mostly intraclass correlations. We analyzed the methods of scale development and reliability estimation for factors that are likely to influence the resulting estimate, including aspects of the judges, the targets, what is observed, and the kind of reliability coefficient.
  
  
  RESULTS
  Typical reliability was approximately 0.7, but estimates were quite varied in magnitude, from quite low (0.3) to quite high (0.9). Thus, highly reliable assessment is possible, but rarely achieved. Factors that are likely to influence the magnitude of reliability include the number of assessors, the participants' range in skill level, and the observational process. Most studies do not present enough information to make the estimates comparable from one study to the next.
  
  
  CONCLUSIONS
  Although nontechnical skills can be measured reliably under some circumstances, it is not clear from the existing literature that one scale provides more reliable data than another. We recommend several practices that would help readers to make informed decisions about which scales are most useful for their purposes.
authors.affiliation: Department of Psychology, Middle East Technical University, Ankara, Turkey.
authors.forename: Ipek
authors.initials: I
authors.lastname: Mete
authors.affiliation: Department of Psychology, University of South Florida, Tampa, Florida. Electronic address: mbrannick@usf.edu.
authors.forename: Michael T
authors.initials: MT
authors.lastname: Brannick
mesh_terms: D002983:Clinical Competence; D004521:Educational Measurement / Q000379:methods*; D013502:General Surgery / Q000193:education*; D006801:Humans; D033183:Interdisciplinary Communication; D010348:Patient Care Team / Q000592:standards*; D015203:Reproducibility of Results
keywords: Interpersonal and Communication Skills; Practice-Based Learning and Improvement; assessment; interprofessional; metrics; outcomes; teamwork