title: Comparing correlated kappas by resampling: is one level of agreement significantly different from another?
abstract: Researchers comparing diagnostic systems or screening tests frequently need to compare indices of agreement such as the kappa coefficient. While asymptotic methods exist for comparing kappas derived from independent samples, no satisfactory approach exists for comparing kappas derived from the same or related samples. Resampling methods for comparing kappa coefficients obtained from the same sample are presented. Easily undertaken using readily available software, these methods are illustrated by application to a small sample of psychiatric data, as well as to several thousand samples of simulated data. An acceptable type I error rate was exhibited. Resampling techniques-easily implemented and making few assumptions-deserve wider application in psychiatric research.
authors.affiliation: Department of Psychological Medicine, Monash University, Victoria, Australia.
authors.forename: D P
authors.initials: DP
authors.lastname: McKenzie
authors.forename: A J
authors.initials: AJ
authors.lastname: Mackinnon
authors.forename: N
authors.initials: N
authors.lastname: Péladeau
authors.forename: P
authors.initials: P
authors.lastname: Onghena
authors.forename: P C
authors.initials: PC
authors.lastname: Bruce
authors.forename: D M
authors.initials: DM
authors.lastname: Clarke
authors.forename: S
authors.initials: S
authors.lastname: Harrigan
authors.forename: P D
authors.initials: PD
authors.lastname: McGorry
mesh_terms: D000328:Adult; D003866:Depressive Disorder / Q000175:diagnosis / Q000523:psychology; D005260:Female; D006801:Humans; D008297:Male; D008432:Mathematical Computing; D009010:Monte Carlo Method; D015588:Observer Variation; D010555:Personality Inventory / Q000706:statistics & numerical data; D011594:Psychometrics / Q000379:methods*; D015203:Reproducibility of Results