title: Determining reliability of clinical assessment scores in real time.
abstract: BACKGROUND
  Assessment score reliability is usually based on a single analysis. However, reliability is an essential component of validity and assessment validation and revision is a never-ending cycle. For ongoing assessments over extended time frames, real-time reliability computations may alert users to possible changes in the learning environment that are revealed by variations in reliability over time.
  
  
  OBJECTIVE
  To develop software that calculates the reliability of clinical assessments in real time.
  
  
  METHODS
  Over 2,400 assessment forms were analyzed. We developed software that calculates reliability in real time. Software accuracy was verified by comparing data from our software with a standard method. Factor analysis determined scale dimensionality.
  
  
  RESULTS
  Correlation between our software and a standard method was excellent (ICC for kappas = 0.97; Cronbach's alphas differed by < 0.03). Cronbach's alpha ranged from 0.94 to 0.97 and weighted kappa ranged from 0.08 to 0.40. Factor analysis confirmed 3 teaching domains.
  
  
  CONCLUSIONS
  We describe an accurate method for calculating reliability in real time. The benefit of real time computation is that it provides a mechanism for detecting possible changes (related to curriculum, teachers, and students) in the learning environment indicated by changes in reliability over time. This technique will enable investigators to monitor and detect changes in the reliability of assessment scores and, with future study, isolate aspects of the learning environment that impact on reliability.
authors.affiliation: Department of Internal Medicine, Mayo Clinic College of Medicine, Rochester, Minnesota 55905, USA. Beckman.Thomas@mayo.edu
authors.forename: Thomas J
authors.initials: TJ
authors.lastname: Beckman
authors.forename: Jayawant N
authors.initials: JN
authors.lastname: Mandrekar
authors.forename: Gregory J
authors.initials: GJ
authors.lastname: Engstler
authors.forename: Robert D
authors.initials: RD
authors.lastname: Ficalora
mesh_terms: D000328:Adult; D002983:Clinical Competence; D004503:Education, Medical, Graduate / Q000592:standards*; D004521:Educational Measurement / Q000592:standards*; D005163:Factor Analysis, Statistical; D005260:Female; D006801:Humans; D007388:Internal Medicine / Q000193:education / Q000592:standards*; D007396:Internship and Residency / Q000592:standards*; D008297:Male; D015203:Reproducibility of Results; D012984:Software