title: Determining agreement using rater characteristics.
abstract: When evaluating the usefulness of clinical information for the diagnosis of disease, multiple raters provide a diagnosis for the same set of data. These ratings provide important insights into the performance of the diagnosis, determining the accuracy of each rater's diagnosis compared to the truth standard and the level of agreement among the raters. We demonstrate that the intraclass correlation coefficient (ICC) is dependent on the sensitivities and specificities of the raters involved in the study. Given the sensitivity and specificity of any number of raters, along with the prevalence of disease, the expected ICC can be determined.
authors.affiliation: a Department of Biostatistics , Teva Pharmaceuticals , West Chester , Pennsylvania , USA.
authors.forename: Mat D
authors.initials: MD
authors.lastname: Davis
authors.affiliation: b Department of Biostatistics , Chiltern International , King of Prussia , Pennsylvania , USA.
authors.forename: Jeffrey
authors.initials: J
authors.lastname: Joseph
mesh_terms: D003627:Data Interpretation, Statistical; D003933:Diagnosis; D006801:Humans; D015588:Observer Variation; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity
keywords: Agreement; intraclass correlation; reliability; sensitivity; specificity