Choice of agreement indices for assessing and improving measurement reproducibility in a core laboratory setting.
Clinical core laboratories, such as Echocardiography core laboratories, are increasingly used in clinical studies with imaging outcomes as primary, secondary, or surrogate endpoints. While many factors contribute to the quality of measurements of imaging variables, an essential step in ensuring the value of imaging data includes formal assessment and control of reproducibility via intra-observer and inter-observer reliability. There are many different agreement/reliability indices in the literature. However, different indices may lead to different conclusions and it is not clear which index is the preferred choice as an overall indication of data quality and a tool for providing guidance on improving quality and reliability in a core lab setting. In this paper, we pre-specify the desirable characteristics of an agreement index for assessing and improving reproducibility in a core lab setting; we compare existing agreement indices in terms of these characteristics to choose a preferred index. We conclude that, among the existing indices reviewed, the coverage probability for assessing agreement is the preferred agreement index on the basis of computational simplicity, its ability for rapid identification of discordant measurements to provide guidance for review and retraining, and its consistent evaluation of data quality across multiple reviewers, populations, and continuous/categorical data.
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA huiman.barnhart@dm.duke.edu.
Huiman X
HX
Barnhart
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Eric
E
Yow
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Anna Lisa
AL
Crowley
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Melissa A
MA
Daubert
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Dawn
D
Rabineau
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Robert
R
Bigelow
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Michael
M
Pencina
Duke Clinical Research Institute, Duke University Medical Center, Durham, USA.
Pamela S
PS
Douglas
D006801:Humans; D007753:Laboratories; D008944:Mitral Valve Insufficiency / Q000175:diagnosis; D015203:Reproducibility of Results; D014463:Ultrasonography / Q000592:standards; D016277:Ventricular Function, Left
agreement; measurement error; method comparison; reliability; reproducibility