Measurement of observer agreement.
Statistical measures are described that are used in diagnostic imaging for expressing observer agreement in regard to categorical data. The measures are used to characterize the reliability of imaging methods and the reproducibility of disease classifications and, occasionally with great care, as the surrogate for accuracy. The review concentrates on the chance-corrected indices, kappa and weighted kappa. Examples from the imaging literature illustrate the method of calculation and the effects of both disease prevalence and the number of rating categories. Other measures of agreement that are used less frequently, including multiple-rater kappa, are referenced and described briefly.
Kundel
Harold L
HL
Department of Radiology and MCP Hahnemann School of Public Health, University of Pennsylvania Medical Center, 3600 Market St, Suite 370, Philadelphia, PA 19104, USA. kundel@rad.upenn.edu
Polansky
Marcia
M
D003627:Data Interpretation, Statistical; D003952:Diagnostic Imaging; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results