title: Interpreting kappa in observational research: baserate matters.
abstract: Kappa (Cohen, 1960) is a popular agreement statistic used to estimate the accuracy of observers. The response of kappa to differing baserates was examined and methods for estimating the accuracy of observers presented. Results suggest that setting a single value of kappa as "minimally acceptable" (i.e., a criterion value) is not useful in ensuring adequate accuracy of observers. Instead, researchers should use the best estimate of the true baserate of the target behavior and the obtained kappa to estimate the accuracy of observers. Investigators can then compare the estimated accuracy of observers to a preselected criterion level. Guidelines are given for selecting a criterion accuracy level.
authors.affiliation: Sonoma State University, Rohner Park, CA 94928-2152, USA. corneliabruckner@sonoma.edu
authors.forename: Cornelia Taylor
authors.initials: CT
authors.lastname: Bruckner
authors.forename: Paul
authors.initials: P
authors.lastname: Yoder
mesh_terms: D001288:Attention; D001321:Autistic Disorder / Q000175:diagnosis* / Q000523:psychology; D002648:Child; D002653:Child Behavior Disorders / Q000175:diagnosis* / Q000523:psychology; D005260:Female; D006801:Humans; D008607:Intellectual Disability / Q000175:diagnosis* / Q000523:psychology; D008297:Male; D015233:Models, Statistical; D019370:Observation; D015588:Observer Variation; D010287:Parent-Child Relations; D010552:Personality Assessment / Q000706:statistics & numerical data*; D011594:Psychometrics / Q000706:statistics & numerical data; D015203:Reproducibility of Results