Relation of observer agreement to accuracy according to a two-receiver signal detection model of diagnosis.
A long-standing problem of medical diagnosis is why rates of observer disagreement are so high. To study this question a model of test interpretation is defined which accounts for disagreement when observers use identical diagnostic strategies. Test subjects are assigned to one of two disease classes on the basis of two parameters: the test score and lambda, the diagnostic certainty, a parameter analogous to signal-to-noise ratio. The rate of disagreement and the mean error rate of independent ideal observers are derived as a function of diagnostic strategy. It follows from the model that: 1) Maximization of the ratio of disagreement to error defines an optimal diagnostic strategy with advantages over maximum likelihood and Bayes diagnostic strategies. 2) Disagreement exceeds error if the sensitivity and specificity of independent observers exceed one-half for all lambda. 3) The ratio of disagreement to error can be no greater than two if test interpretation is dichotomous but can approach infinity for trichotomous interpretation, i.e., if observers are allowed to interpret cases as indeterminate.
Quinn
M F
MF
Department of Radiology, University of Southern California, Los Angeles.
D001499:Bayes Theorem; D019937:Diagnostic Techniques and Procedures; D006801:Humans; D015233:Models, Statistical; D015203:Reproducibility of Results; D012680:Sensitivity and Specificity