title: Assessing observer agreement in studies involving replicated binary observations.
abstract: We propose a new approach to evaluating agreement between two observers making binary observations on a set of subjects. This approach compares the probability of disagreement (or discordance) between the observers to the probability of disagreement between replicated observations made by the same observer on the same subject. We consider two situations: (1) a symmetric assessment of agreement between two observers, and (2) an assessment of the agreement of a new observer with an imperfect "gold standard". We develop a nonparametric method for estimation of the new agreement coefficients when observers make replicated readings on each subject. The reliability of the estimation method is examined via a simulation study. Data from a study aimed at determining the validity of diagnosis of breast cancer based on mammograms is used to illustrate the new concepts and methods.
authors.affiliation: Department of Biostatistics, Rollins School of Public Health, Emory University, Atlanta, Georgia 30322, USA. mhaber@sph.emory.edu
authors.forename: Michael
authors.initials: M
authors.lastname: Haber
authors.forename: Jingjing
authors.initials: J
authors.lastname: Gao
authors.forename: Huiman X
authors.initials: HX
authors.lastname: Barnhart
mesh_terms: D000465:Algorithms; D001943:Breast Neoplasms / Q000175:diagnosis*; D003198:Computer Simulation; D019937:Diagnostic Techniques and Procedures / Q000706:statistics & numerical data*; D005260:Female; D006801:Humans; D008327:Mammography / Q000706:statistics & numerical data; D015233:Models, Statistical; D015588:Observer Variation; D011336:Probability; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity; D018709:Statistics, Nonparametric