title: Method matters: Understanding diagnostic reliability in DSM-IV and DSM-5.
abstract: Diagnostic reliability is essential for the science and practice of psychology, in part because reliability is necessary for validity. Recently, the DSM-5 field trials documented lower diagnostic reliability than past field trials and the general research literature, resulting in substantial criticism of the DSM-5 diagnostic criteria. Rather than indicating specific problems with DSM-5, however, the field trials may have revealed long-standing diagnostic issues that have been hidden due to a reliance on audio/video recordings for estimating reliability. We estimated the reliability of DSM-IV diagnoses using both the standard audio-recording method and the test-retest method used in the DSM-5 field trials, in which different clinicians conduct separate interviews. Psychiatric patients (N = 339) were diagnosed using the SCID-I/P; 218 were diagnosed a second time by an independent interviewer. Diagnostic reliability using the audio-recording method (N = 49) was "good" to "excellent" (M κ = .80) and comparable to the DSM-IV field trials estimates. Reliability using the test-retest method (N = 218) was "poor" to "fair" (M κ = .47) and similar to DSM-5 field-trials' estimates. Despite low test-retest diagnostic reliability, self-reported symptoms were highly stable. Moreover, there was no association between change in self-report and change in diagnostic status. These results demonstrate the influence of method on estimates of diagnostic reliability.
authors.affiliation: Department of Psychology.
authors.forename: Michael
authors.initials: M
authors.lastname: Chmielewski
authors.affiliation: Department of Psychology, University of Notre Dame.
authors.forename: Lee Anna
authors.initials: LA
authors.lastname: Clark
authors.affiliation: Department of Psychology, University of Toronto.
authors.forename: R Michael
authors.initials: RM
authors.lastname: Bagby
authors.affiliation: Department of Psychology, University of Notre Dame.
authors.forename: David
authors.initials: D
authors.lastname: Watson
mesh_terms: D000293:Adolescent; D000328:Adult; D000368:Aged; D000369:Aged, 80 and over; D003937:Diagnosis, Differential; D039721:Diagnostic and Statistical Manual of Mental Disorders; D005260:Female; D006801:Humans; D008297:Male; D001523:Mental Disorders / Q000175:diagnosis*; D008875:Middle Aged; D015203:Reproducibility of Results; D055815:Young Adult