title: A method for measuring interrater agreement on checklists.
abstract: A method for measuring interrater agreement on checklists is presented. This technique does not assign individual scores to raters, but computes a single agreement score from the concordance of their check mark configurations. An overall coefficient of agreement, called phi, is derived. The agreement coefficient that is expected by chance and the statistical significance of phi are determined by statistical simulation. Despite the dichotomous nature of the checklist agreement (raters either agree or disagree on items), we show that the binomial distribution does not provide a means for testing the statistical significance of phi. A medical education study is used to illustrate the phi methodology.
authors.affiliation: University of Illinois at Chicago, Department of Family Medicine 60612-7248, USA. Sinacore@uic.edu
authors.forename: J M
authors.initials: JM
authors.lastname: Sinacore
authors.forename: K J
authors.initials: KJ
authors.lastname: Connell
authors.forename: A J
authors.initials: AJ
authors.lastname: Olthoff
authors.forename: M H
authors.initials: MH
authors.lastname: Friedman
authors.forename: M R
authors.initials: MR
authors.lastname: Gecht
mesh_terms: D004504:Education, Medical, Undergraduate / Q000379:methods*; D004521:Educational Measurement / Q000379:methods*; D006801:Humans; D015588:Observer Variation; D016544:Patient Simulation; D010808:Physical Examination; D015397:Program Evaluation / Q000379:methods*; D015203:Reproducibility of Results; D013223:Statistics as Topic