title: The Rater Applied Performance Scale: development and reliability.
abstract: Previous studies of rater performance and interrater reliability have used passive scoring tasks such as rating patients from a videotaped interview. Little is known, however, about how well raters conduct assessments on real patients or how reliably they apply scoring criteria during actual assessment sessions. With growing recognition of the importance of monitoring and review of actual evaluation sessions, there is need for a systematic approach to quantify raters' applied performance. The Rater Applied Performance Scale (RAPS) measures six dimensions of rater performance (adherence, follow-up, clarification, neutrality, rapport, and accuracy) based on reviews of audiotaped or videotaped assessment sessions or on live monitoring of assessment sessions. We tested this new scale by having two reviewers rate 20 Hamilton Depression Scale rating sessions ascertained from a multi-site depression trial. We found good internal consistency for the RAPS. Interrater (i.e. inter-reviewer) reliability was satisfactory for RAPS total score ratings. In addition, RAPS ratings correlated with quantitative measures of scoring accuracy based on independent expert ratings. Preliminary psychometric data suggest that the RAPS may be a valuable tool for quantifying the performance of clinical raters. Potential applications of the RAPS are considered.
authors.affiliation: Anxiety Clinic #69, New York State Psychiatric Institute/College of Physicians and Surgeons, Columbia University, 1051 Riverside Drive, NY 10032, USA. Lipsitz@pi.cpmc.columbia.edu
authors.forename: Joshua
authors.initials: J
authors.lastname: Lipsitz
authors.forename: Ken
authors.initials: K
authors.lastname: Kobak
authors.forename: Alan
authors.initials: A
authors.lastname: Feiger
authors.forename: Dawn
authors.initials: D
authors.lastname: Sikich
authors.forename: Georges
authors.initials: G
authors.lastname: Moroz
authors.forename: Ama
authors.initials: A
authors.lastname: Engelhard
mesh_terms: D003865:Major Depressive Disorder / Q000175:diagnosis* / Q000453:epidemiology*; D005500:Follow-Up Studies; D006801:Humans; D015588:Observer Variation; D011594:Psychometrics / Q000379:methods; D015203:Reproducibility of Results; D011795:Surveys and Questionnaires; D014743:Videotape Recording