title: An examination of the comparative reliability, validity, and accuracy of performance ratings made using computerized adaptive rating scales.
abstract: This laboratory research compared the reliability, validity, and accuracy of a computerized adaptive rating scale (CARS) format and 2 relatively common and representative rating formats. The CARS is a paired-comparison rating task that uses adaptive testing principles to present pairs of scaled behavioral statements to the rater to iteratively estimate a ratee's effectiveness on 3 dimensions of contextual performance. Videotaped vignettes of 6 office workers were prepared, depicting prescripted levels of contextual performance, and 112 subjects rated these vignettes using the CARS format and one or the other competing format. Results showed 23%-37% lower standard errors of measurement for the CARS format. In addition, validity was significantly higher for the CARS format (d = .18), and Cronbach's accuracy coefficients showed significantly higher accuracy, with a median effect size of .08. The discussion focuses on possible reasons for the results.
authors.affiliation: Personnel Decisions Research Institutes and Department of Psychology, University of South Florida, Tampa 33602, USA. wally.borman@personneldecisions.com
authors.forename: W C
authors.initials: WC
authors.lastname: Borman
authors.forename: D E
authors.initials: DE
authors.lastname: Buck
authors.forename: M A
authors.initials: MA
authors.lastname: Hanson
authors.forename: S J
authors.initials: SJ
authors.lastname: Motowidlo
authors.forename: S
authors.initials: S
authors.lastname: Stark
authors.forename: F
authors.initials: F
authors.lastname: Drasgow
mesh_terms: D000328:Adult; D003198:Computer Simulation; D001330:Electronic Data Processing; D004650:Employee Performance Appraisal; D005260:Female; D006801:Humans; D008297:Male; D010560:Personnel Selection; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D013647:Task Performance and Analysis; D014741:Video Recording