title: Sources of unreliability in depression ratings.
abstract: BACKGROUND
  Good interrater reliability is essential to minimize error variance and improve study power. Reasons why raters differ in scoring the same patient include information variance (different information obtained because of asking different questions), observation variance (the same information is obtained, but raters differ in what they notice and remember), interpretation variance (differences in the significance attached to what is observed), criterion variance (different criteria used to score items), and subject variance (true differences in the subject). We videotaped and transcribed 30 pairs of interviews to examine the most common sources of rater unreliability.
  
  
  METHODS
  Thirty patients who experienced depression were independently interviewed by 2 different raters on the same day. Raters provided rationales for their scoring, and independent assessors reviewed the rationales, the interview transcripts, and the videotapes to code the main reason for each discrepancy. One third of the interviews were conducted by raters who had not administered the Hamilton Depression Rating Scale before; one third, by raters who were experienced but not calibrated; and one third, by experienced and calibrated raters.
  
  
  RESULTS
  Experienced and calibrated raters had the highest interrater reliability (intraclass correlation [ICC]; r = 0.93) followed by inexperienced raters (r = 0.77) and experienced but uncalibrated raters (r = 0.55). The most common reason for disagreement was interpretation variance (39%), followed by information variance (30%), criterion variance (27%), and observation variance (4%). Experienced and calibrated raters had significantly less criterion variance than the other cohorts (P = 0.001).
  
  
  CONCLUSIONS
  Reasons for disagreement varied by level of experience and calibration. Experienced and uncalibrated raters should focus on establishing common conventions, whereas experienced and calibrated raters should focus on fine tuning judgment calls on different thresholds of symptoms. Calibration training seems to improve reliability over experience alone. Experienced raters without cohort calibration had lower reliability than inexperienced raters.
authors.affiliation: MedAvante Research Institute, Madison, WI, USA. kkobak@medavante.net
authors.forename: Kenneth A
authors.initials: KA
authors.lastname: Kobak
authors.forename: Brianne
authors.initials: B
authors.lastname: Brown
authors.forename: Ian
authors.initials: I
authors.lastname: Sharp
authors.forename: Hollie
authors.initials: H
authors.lastname: Levy-Mack
authors.forename: Kurrie
authors.initials: K
authors.lastname: Wells
authors.forename: Felice
authors.initials: F
authors.lastname: Ockun
authors.forename: Felice
authors.initials: F
authors.lastname: Okum
authors.forename: Janet B W
authors.initials: JB
authors.lastname: Williams
mesh_terms: D002983:Clinical Competence; D003866:Depressive Disorder / Q000175:diagnosis* / Q000453:epidemiology* / Q000523:psychology; D006801:Humans; D007407:Interviews as Topic; D015588:Observer Variation; D011361:Professional Competence; D011369:Professional-Patient Relations; D011569:Psychiatric Status Rating Scales; D011594:Psychometrics; D015203:Reproducibility of Results; D011795:Surveys and Questionnaires; D014743:Videotape Recording